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This  report  responds  to  your  request  that  we  review  the  Air  Force’s  actions  to  complete  its 
satellite  command  and  control  system  upgrade,  called  the  Command  and  Control  Segment 
(ccs),  and  turn  it  over  to  the  Air  Force  Space  Command  in  April  1993.  We  found  that  the  Air 
Force  may  not  be  able  to  complete  ccs  on  schedule,  and  made  recommendations  to  help 
ensure  that  CCS  provides  the  needed  capacity  and  performance  when  completed. 


We  are  sending  copies  of  this  report  to  the  Chairmen,  Senate  Committees  on  Appropriations 
and  Governmental  Affairs;  House  Committees  on  Appropriations,  Armed  Services,  and 
Government  Operations;  the  Director,  Office  of  Management  and  Budget;  and  the  Secretaries 
of  Defense  and  the  Air  Force.  We  are  also  making  copies  available  to  others  upon  request. 


This  report  was  prepared  under  the  direction  of  Samuel  W.  Bowlin,  Director,  Defense  and 
Security  Information  Systems,  who  can  be  reached  at  (202)  336-6240.  Other  major 
contributors  are  listed  in  appendix  III. 


Assistant  Comptroller  General 


A 


■  T  STATEMENT  '  j 


’07r -J  I'-t  public  release)  j' 
Pis! r ib-iiicn  f 


Executive  Summary 


Purpose 


The  Department  of  Defense’s  (dod)  satellites  perform  a  wide  variety  of 
missions  to  support  national  security  interests  and  pursue  advances  in 
research  and  development.  The  missions  include  communications,  navi¬ 
gation,  strategic  defense,  tactical  warning  and  attack  assessment,  and 
other  classified  missions.  The  Air  Force’s  Satellite  Control  Network  con¬ 
trols  satellite  launches,  maintains  satellites,  and  keeps  them  in  orbit. 
Since  1980,  Air  Force  Systems  Command  has  been  upgrading  its  satellite 
command  and  control  system,  called  the  Command  and  Control  Segment 
(ccs).  The  Chairman,  Subcommittee  on  Strategic  Forces  and  Nuclear 
Deterrence,  Senate  Committee  on  Armed  Services,  concerned  about  the 
program’s  shifting  requirements,  continuing  schedule  delays,  technical 
problems,  and  increasing  costs,  asked  gao  to  review  the  Air  Force’s 
actions  to  complete  the  command  and  control  system  upgrade  and  turn 
it  over  to  Air  Force  Space  Command  in  1993. 


Background 


Since  the  late  1960s,  the  Air  Force  has  used  the  Current  Data  System 
(CDS)  to  cti.  mand  and  control  its  satellites.  However,  cds  uses  obsolete 
computers  and  is  expensive  to  maintain,  with  maintenance  costs  of 
$30  million  annually,  ccs  was  planned  as  a  modem  system  that  could 
replace  CDS,  as  well  as  cut  costs,  improve  performance,  and  enhance 
functionality.  Until  ccs  is  completed,  the  Air  Force  is  using  both  the  par¬ 
tially  completed  ccs  and  the  old  cds.  ccs  is  not  scheduled  to  be  com¬ 
pleted,  and  cds  deactivated,  until  July  1993. 

The  Air  Force  runs  ccs  in  1 1  computer  centers  called  mission  control 
complexes  (mccs).  An  mcc  supports  satellite  launches  and  orbiting  satel¬ 
lites,  including  tracking  and  determining  how  well  each  is  working.  Each 
mcc  controls  a  specific  type  and  number  of  satellites.  Collectively,  the 
mccs  control  about  72  satellites. 

Systems  Command  and  Space  Command  share  responsibility  for  com¬ 
manding  and  controlling  the  network’s  satellites  through  their  respec¬ 
tive  mccs.  Currently,  Systems  Command  uses  both  cds  and  the  current 
version  of  ccs  to  command  and  control  its  satellites.  Space  Command  is 
gradually  assuming  responsibility  for  operational  military  satellites  and 
uses  only  the  current  version  of  ccs  to  command  and  control  these  satel¬ 
lites.  Once  Systems  Command  completes  the  ccs  upgrade  and  the  system 
is  fully  operational,  responsibility  for  commanding  and  controlling  those 
operational  satellites  will  transfer  to  Space  Command  and  cds  will  be 
deactivated. 
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Results  in  Brief 


The  Air  Force’s  ability  to  meet  the  July  1993  deadline  is  questionable 
and  depends  largely  on  whether  it  can  fix  ccs’  critical  operational  defi¬ 
ciencies.  However,  slow  progress  in  correcting  some  of  these  problems, 
combined  with  the  growing  number  of  satellites  ccs  is  expected  to  con¬ 
trol  and  the  new  problems  that  continue  to  be  found,  increases  the  risk 
that  ccs  will  not  be  completed  on  time. 

Further,  ccs  may  not  be  able  to  provide  the  needed  capacity  and  per¬ 
formance  because  the  Air  Force  has  not:  adequately  defined  work-load 
requirements  for  ccs,  adequately  tested  ccs,  set  up  an  effective  capacity 
and  performance  management  program,  or  obtained  adequate  software 
documentation.  Without  up-to-date  work-load  requirements,  the  Air 
Force  cannot  (1)  assure  that  the  system  is  properly  sized  for  current  and 
future  needs,  (2)  develop  tests  to  adequately  stress  the  system,  and 
(3)  reliably  assess  the  impact  of  ccs  changes  on  mission  effectiveness. 
Without  adequate  testing,  the  Air  Force  cannot  determine  if  ccs  will 
effectively  support  all  satellites.  Without  an  effective  capacity  and  per¬ 
formance  management  program,  the  Air  Force  does  not  know  how  well 
CCS  is  working,  nor  can  it  effectively  plan  for  future  computer  resource 
requirements.  And  finally,  unless  the  Air  Force  corrects  deficiencies  in 
ccs’  software  documentation,  it  will  be  increasingly  difficult  to  maintain 
and  enhance  ccs  as  requirements  continue  to  change  and  evolve. 

If  these  problems  are  not  resolved  quickly,  the  Air  Force  will  be  forced 
to  continue  using  CDS  to  perform  some  command  and  control  functions, 
and  spending  $30  million  annually  to  maintain  CDS,  which  is  recognized 
as  being  outdated. 


Principal  Findings 


CCS  Cannot  Perform  Key 
Command  and  Control 
Operations 


ccs  is  required  to  be  able  to  contact1  and  communicate  with  multiple 
satellites  simultaneously  under  expected  peak  work-load  conditions.  At 
most  mccs,  however,  ccs  cannot  make  the  required  number  of  simulta¬ 
neous  contacts.  Making  simultaneous  contacts  is  a  key  requirement  used 
to  judge  whether  ccs  is  ready  to  handle  all  satellite  control  functions  at 
an  mcc,  and  to  permit  CDS  to  be  deactivated.  In  addition,  ccs  has  other 


'A  contact  is  the  reception  and/or  transmission  of  information  between  a  space  vehicle  and  an  MCC. 
There  are  three  primary  functions  of  a  contact:  telemetry,  tracking,  and  commanding. 
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serious  operational  problems  (e.g.,  long  recovery  time  from  system  fail¬ 
ures).  Some  problems  affect  all  satellites  and  mccs,  others  affect  only  a 
specific  satellite  or  mcc.  While  the  Air  Force  has  fixed  several  hundred 
problems,  other  critical  problems  have  not  been  fixed  and  new  problems 
continue  to  be  found.  Meanwhile,  the  costs  to  make  ccs  fully  operational 
continue  to  increase,  gao  estimates  that  the  cost  to  develop  ccs  and  pro¬ 
vide  sustaining  engineering  is  at  least  $906  million  through  September 
1991. 


Work-Load  Requirements 
Not  Adequately  Defined 


Well-defined  work-load  requirements  are  needed  to  help  size  the  system 
for  current  and  future  needs,  to  develop  tests  to  stress  the  system,  and 
to  ensure  that  the  system  functions  effectively  under  expected  peak 
work  loads.  However,  the  work-load  requirements  for  the  mccs  are  out¬ 
dated  and  do  not  accurately  describe  expected  peak  work  loads.  The 
work-load  requirements  were  last  updated  in  1987.  Since  that  time,  the 
number  of  satellites  has  increased  by  36  percent  and,  according  to  Sys¬ 
tems  Command’s  Program  Office  officials,  the  mccs  now  handle  dif¬ 
ferent  types  and  numbers  of  satellites.  Because  the  work-load 
requirements  have  not  been  updated,  they  do  not  reflect  actual  opera¬ 
tional  conditions  and  cannot  be  used  effectively  to  size,  stress,  and  test 
the  system. 


CCS  Testing  Inadequate  Without  accurately  defined  work  load  requirements,  the  Air  Force 

cannot  adequately  test  and  evaluate  ccs  to  determine  if  it  meets  the 
mccs’  requirements  before  being  turned  over  to  Space  Command.  Fur¬ 
ther,  new  ccs  software  releases  are  not  being  adequately  tested  in  the 
Air  Force’s  Software  Development  Testing  Laboratory  before  being 
used.  Therefore,  critical  deficiencies  are  not  detected  until  the  software 
is  actually  used  to  control  satellites.  Between  Marcn  1989  and  March 
1990,  three  major  software  releases  were  tested  m  the  laboratory;  no 
critical  deficiencies  were  found.  However,  after  these  releases  were 
approved  for  use  in  the  mccs,  approximately  175  critical  deficiencies 
were  discovered.  At  this  late  stage,  these  problems  are  more  difficult 
and  expensive  to  correct.  Moreover,  finding  a  critical  problem  while  ccs 
is  helping  launch  or  track  a  satellite  could  jeopardize  that  satellite. 

These  software  deficiencies  are  due,  at  least  in  part,  to  the  fact  that  the 
laboratory  does  not  have  the  hardware  needed  to  simulate  the  mccs’ 
high-stress  work  loads.  The  laboratory  can  only  test  whether  new 
releases  can  make  three  simultaneous  contacts.  However,  because  the 
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mccs  use  the  systems  under  tougher  conditions,  e.g.,  five  simultaneous 
contacts,  critical  deficiencies  are  discovered. 

Capacity  and  Performance 
Management  Program 
nadequate 

In  order  to  measure  computer  use  and  performance  and  to  help  predict 
future  needs,  agencies  should  routinely  collect  and  analyze  detailed 
capacity  and  performance  data.  However,  the  Air  Force  is  not  doing  this 
and,  as  a  result,  does  not  know  how  well  ccs  is  working  and  how  much 
capacity  is  being  used.  Without  this  information,  the  Air  Force  cannot 
effectively  determine  whether  and  when  changes  are  needed  to  meet 
mission  requirements. 

Instead,  the  Air  Force  relies  on  three  sources  of  information  to  manage 
ccs  computer  resources:  (1)  data  on  satellite  contact  success  rates  using 
ccs  and  CDS,  (2)  computer  operators’  perceptions  of  ccs’  limitations,  and 
(3)  infrequent  ad  hoc  analyses  of  computer  capacity  and  performance. 
While  these  provide  some  useful  information,  they  do  not  give  a  com¬ 
plete  picture  of  computer  performance,  mostly  because  they  do  not  mea¬ 
sure  actual  use  or  continuously  assess  performance.  Furthermore,  they 
do  not  offer  the  careful,  comprehensive  analysis  needed  to  manage  a 
system  this  large  and  complex. 

Software  Poorly 
documented 

Computer  system  software  documentation  must  be  well-organized,  com¬ 
plete,  and  up-to-date,  so  that  systems  analysts  can  understand  the 
system,  identify  problems,  and  make  needed  changes,  ccs’  software  doc¬ 
umentation,  however,  is  incomplete,  out-of-date,  and  difficult  to  use 

Poor  documentation  delays  fixing  problems  and  increases  maintenance 
costs,  which  make  up  the  largest  percentage  of  a  system’s  life  cycle 
costs. 

Recommendations 

gao  recommends  that  the  Secretary  of  Defense  direct  the  Secretary  of 
the  Air  Force  to  ensure  that  (1)  ccs  work-load  requirements  (specifi¬ 
cally,  the  peak  work-load  requirements  for  each  mcc’s  computer  system) 
are  immediately  updated  and  kept  current;  (2)  the  updated  work-load 
requirements  are  used  to  operationally  test  ccs  at  the  mccs,  and  stress 
test  each  ccs  software  release  before  it  is  used  in  the  mccs;  (3)  a  compre¬ 
hensive  ccs  capacity  and  performance  management  program  is  immedi¬ 
ately  established;  and  (4)  ccs’  software  documentation  is  adequately 
prepared. 
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Vgency  Comments 


dod  stated  it  partially  concurred  with  most  of  the  findings  and  all  of  the 
recommendations,  and  has  taken  steps  to  address  each  recommendation. 
dod  stated  that  the  Air  Force  has  defined  ccs  work-load  requirements 
and  plans  to  use  them  to  operationally  test  ccs  at  the  mccs.  Further,  dod 
stated  the  Air  Force  (1)  has  identified  requirements  to  upgrade  ccs 
testing  capabilities,  (2)  will  assess  its  capacity  and  performance  manage¬ 
ment  efforts,  and  (3)  is  upgrading  some  of  ccs’  software  documentation 
However,  dod  did  not  require  the  Air  Force  to  establish  a  formal 
capacity  and  performance  management  plan,  or  ensure  that  ccs  is  stress 
tested  before  it  is  used  in  the  mccs.  These  actions  are  essential  to  assure 
ccs  effectively  and  efficiently  meets  its  mission  requirements. 

dod  disagreed  with  gao’s  finding  that  ccs  costs  are  continuing  to 
increase,  dod  believes  that  gao  incorrectly  combined  acquisition  costs 
(associated  with  the  completed  development  of  ccs)  with  subsequent  life 
cycle  costs.  Since  the  purpose  of  ccs  was  to  replace  the  old,  out-of-date 
CDS,  gao  believes  it  is  appropriate  to  report  all  costs  until  all  original 
requirements  for  ccs  are  completed  and  CDS  is  deactivated.  An  evalua¬ 
tion  of  dod’s  comments  is  included  in  chapter  4  and  appendix  II. 
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apter  1 _ 

itroduction 


The  Department  of  Defense’s  (dod)  satellites  perform  a  wide  variety  of 
missions  to  support  national  security  interests  and  pursue  advances  in 
research  and  development.  The  missions  include  communications,  navi¬ 
gation,  strategic  defense,  tactical  warning  and  attack  assessment,  and 
other  classified  missions.  By  the  year  2015,  the  Air  Force  Satellite  Con¬ 
trol  Network  (afscn)  is  expected  to  handle  122  Defense  satellites,  com¬ 
pared  to  the  72  Defense  satellites  it  currently  controls.  Therefore,  the 
Air  Force  must  ensure  that  adequate  computer  capability  is  available  to 
command  and  control  these  added  satellites. 

The  Air  Force  has  been  replacing  the  Current  Data  System  (CDS),  an  out¬ 
dated  satellite  command  and  control  system,  with  a  system  called  the 
Command  and  Control  Segment  (ccs).1  ccs  was  expected  to  increase  per¬ 
formance  and  cut  costs  by  replacing  obsolete  computers,  centralizing 
real-time  data  processing,  simplifying  operations,  and  providing  rede¬ 
signed  software  so  that  mission  controllers  could  use  the  system  on  a 
real-time  basis.2 


ission:  Satellite 
ipport 


ccs  is  part  of  afscn,  which  supports  Defense  spacecraft  during  pre¬ 
launch,  launch,  and  while  the  craft  is  in  orbit.3  This  support  includes 
determining  and  tracking  a  satellite’s  orbit,  acquiring  and  processing 
telemetry  data  to  determine  spacecraft  health  and  status,4  receiving  and 
relaying  mission  data  to  users,  generating  and  transmitting  commands, 
and  fixing  operational  anomalies.5  The  network  consists  primarily  of 
worldwide,  ground-based  tracking  stations;  a  test  center  and  an  opera¬ 
tions  center;  dedicated  control  stations;  and  communications  links  con¬ 
necting  these  components.  Appendix  I  illustrates  the  major  elements  of 
AFSCN. 


'The  Command  and  Control  Segment  was  initially  developed  by  the  Data  System  Modernization  pro¬ 
gram  and  was  referred  to  by  that  name  in  our  previous  report,  Military  Space  Operations:  Operational 
l*roblems  Continue  With  the  Satellite  Control  Computer  System  (GAO/IMTEC-89-56,  Aug.  8.  1989). 

2Keal-timc  refers  to  a  system's  capability  to  obtain  data  from  an  activity  or  process,  perform  compu¬ 
tations,  and  return  a  response  quickly  enough  to  affect  the  outcome  of  that  activity  or  process. 

3 AFSCN  also  supports  National  Aeronautics  and  Space  Administration,  North  Atlantic  Treaty  Organi¬ 
zation,  and  other  space  missions. 

4  A  satellite  contains  telemetry  equipment  that  is  used  to  transmit  information  to  the  ground  about  a 
satellite's  position  and  status. 

r’An  anomaly  is  an  unexpected  event  caused  by  a  system  malfunction  or  by  space  environmental 
effects.  If  the  anomaly  is  not  corrected  quickly,  the  satellite  or  its  capabilities  may  be  lost. 
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ccs  is  being  installed  in  the  network’s  two  satellite  command  and  control 
centers:  the  Consolidated  Space  Test  Center  at  Onizuka  Air  Force  Base, 
in  Sunnyvale,  California,  and  the  Consolidated  Space  Operations  Center 
at  Falcon  Air  Force  Base,  near  Colorado  Springs,  Colorado.  These  cen¬ 
ters  are  operated  by  Air  Force  Systems  Command  and  Air  Force  Space 
Command,  respectively.  There  are  eight  mission  control  complexes 
(mccs)  at  the  test  center  and  three  mccs  at  the  operation  center.'5  Each 
mcc  commands  and  controls  specific  types  ar^  numbers  of  satellites; 
each  supports  satellite  launches,  check:;  satellite  operations,  corrects 
prob'ems,  and  keeps  satellites  in  proper  orbit. 

While  ccs’  upgrade  is  underway,  the  completed  portions  are  being  used 
to  control  some  satellites.  Other  satellites  are  still  controlled  by  the  old 
system,  CDS.7  Once  ccs  becomes  fully  operational  at  both  centers,  CDS  will 
be  deactivated.  At  that  time,  the  operations  center  will  mostly  support 
working  military  satellites,  and  the  test  center  will  mostly  support 
research  and  development  programs,  as  well  as  selected  Defense  and 
other  agency  satellite  programs. 


etwork 

esponsibilities 


Systems  Command  and  Space  Command  share  responsibility  for  afscn 
and  ccs.  Initially,  Systems  Command  was  fully  responsible  for  estab¬ 
lishing  user  requirements,  designing  and  developing  ccs,  and  operating 
afscn.  However,  the  responsibility  for  determining  user  requirements 
shifted  to  Space  Command  in  October  1987.  Within  Systems  Command, 
the  Satellite  Control  and  Data  Handling  System  Program  Office  (here¬ 
after  referred  to  as  the  Program  Office)  is  the  network  system  engineer, 
responsible  for  developing  and  acquiring  network  components,  including 
ccs.  Within  Space  Command,  the  Second  Space  Wing  is  the  network 
manager  and  is  responsible  for  commanding  and  controlling  its  opera¬ 
tional  satellites. 

Systems  Command  and  Space  Command  share  responsibility  for  com¬ 
manding  and  controlling  the  network’s  satellites  through  their  respec¬ 
tive  mccs.  Currently,  Systems  Command  uses  both  CDs  and  the  current 
version  of  ccs  to  command  and  control  its  satellites.  Space  Command  has 
been  assuming  responsibility  for  operational  military  satellites  and  uses 
only  the  current  version  of  ccs  to  command  and  control  these  satellites. 


('A  fourth  MCC  is  being  set  up  at  Falcon  Air  Force  Base. 

"For  example,  one  satellite  program  requires  the  processing  of  a  large  amount  of  information  at  an 
extremely  high  speed.  Currently,  only  CDS  can  meet  this  requirement,  although  CCS  is  being  modified 
in  order  to  provide  this  capability  in  the  future 
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Systems  Command  plans  to  transfer  full  responsibility  for  its  seven 
operational  satellite  programs  to  Space  Command  by  July  1993.  The 
transfer  is  being  done  gradually  as  ccs  is  capable  of  supporting  satellite 
programs  at  Space  Command’s  mccs.  As  of  July  1991,  six  satellite  pro¬ 
grams  had  been  transferred  to  Space  Command. 


Objectives,  Scope,  and 
Methodology 


At  the  request  of  the  Chairman,  Subcommittee  on  Strategic  Forces  and 
Nuclear  Deterrence,  Senate  Committee  on  Armed  Services,  and  in  subse¬ 
quent  discussions  with  subcommittee  staff,  we  reviewed  the  Air  Force’s 
actions  to  complete  ccs’  upgrade  and  turn  it  over  to  Space  Command  in 
1993. 


We  interviewed  Air  Force  officials  responsible  for  developing  and  oper¬ 
ating  ccs  and  reviewed  Air  Force  management,  technical,  and  contrac¬ 
tual  documents  to  obtain  information  on  ccs  requirements,  testing, 
capacity  management,  and  software  documentation.  We  analyzed  ccs 
performance  reports,  problem  reports,  cost  and  schedule  data,  and  dis¬ 
cussed  our  findings  with  Systems  Command,  Space  Command,  and  Air 
Force  Operational  Test  and  Evaluation  Center  (afotec)  officials,  (afotec 
is  responsible  for  testing  the  operational  capabilities  of  Space  Com¬ 
mand’s  mccs,  including  ccs  software.)  We  also  interviewed  Air  Force 
headquarters  officials  about  capacity  and  computer  performance. 

To  identify  critei*’  for  effective  system  requirements  and  capacity  man¬ 
agement  programs,  we  reviewed  the  Military  Standard  for  Configuration 
Management  Practices  for  Systems,  Equipment,  Munitions,  and  Com¬ 
puter  Programs  (MIL-STD-483A),  and  the  Military  Standard  for  Specifi¬ 
cation  Practices  (MIL-STD-490A).  We  also  used  the  Federal  Information 
Resources  Management  Regulation,  the  National  Bureau  of  Standards 
work-load  analysis  and  forecasting  guidelines  in  Federal  Information 
Processing  Standards  Publication  49,  and  the  General  Service  Adminis¬ 
tration’s  Federal  Systems  Integration  and  Management  Center  guidelines 
on  capacity  management.  We  compared  these  criteria  to  ccs’  require¬ 
ments  and  the  Air  Force’s  capacity  management  program. 

We  reviewed  the  Air  Force’s  plans  to  test  ccs  at  Falcon  Air  Force  Base 
before  turning  over  responsibility  at  that  site  to  Space  Command.  We 
gathered  information  on  the  number  and  type  of  software  and  hardware 
problems  that  were  identified  after  ccs  software  releases  were  used 
operationally  and  interviewed  Space  Command,  Systems  Command,  and 
afotec  officials  to  discuss  the  testing  and  the  significance  of  the 
problems  we  found. 
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We  reviewed  a  pot  EC’s  software  documentation  evaluations  to  determine 
the  extent  and  significance  of  ccs’  software  documentation  problems.  We 
obtained  the  views  of  apotec,  Space  Command,  and  Systems  Command 
officials  on  the  problems  apotec  identified,  and  determined  what  actions 
were  taken  to  correct  these  problems. 

Our  work  was  conducted  at  Air  Force  headquarters,  in  Washington, 

D.C.;  Space  Systems  Division  at  Los  Angeles  Air  Force  Base,  California; 
Air  Force  Space  Command  headquarters,  Peterson  Air  Force  Base,  Colo* 
rado;  Space  Command’s  Second  Space  Wing,  Falcon  Air  Force  Base,  Col¬ 
orado;  and  Space  Systems  Division’s  Consolidated  Space  Test  Center, 
Onizuka  Air  Force  Base,  California. 

We  performed  our  review  from  November  1989  through  November 
1991,  in  accordance  with  generally  accepted  auditing  standards,  dod 
provided  written  comments  on  a  draft  of  this  report.  These  comments 
and  our  evaluation  are  summarized  in  chapter  4.  Appendix  II  contains 
complete  agency  comments,  as  well  as  our  detailed  evaluation  of  those 
comments. 
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CCS  Implementation — Progress  Being  Made  But 
Costs  Continue  To  Increase  and  Some  Major 
Problems  Still  Need  to  Be  Fixed 


Since  our  1989  report,1  the  Air  Force  has  made  considerable  progress  in 
upgrading  ccs.  The  system  is  now  handling  a  larger  portion  of  satellite 
contacts  and  several  hundred  critical  design  deficiencies  have  been 
fixed.  At  the  same  time,  however,  the  old  CDS  system  is  still  needed  and 
the  cost  to  complete  ccs  continues  to  increase,  requirements  continue  to 
expand,  and  new  system  deficiencies  are  being  found,  ccs  still  cannot 
perform  the  required  number  of  simultaneous  contacts  during  peak 
work-loads  and  the  scheduled  fix  dates  for  some  critical  design  deficien¬ 
cies  have  slipped  over  a  year.  Until  resolved,  ccs  will  not  be  able  to  func¬ 
tion  as  intended  and  the  Air  Force  will  have  to  use  cds  to  augment  it. 


CCS  Costs 
Increase 


Conti  ni  IP  to  The  cost  to  make  ccs  fully  operational  is  increasing.  On  the  basis  of  the 
Air  Force’s  figures  as  of  March  1989  and  an  anticipated  June  1991  com¬ 
pletion  date,  we  estimated  in  1989  that  the  cost  to  develop  ccs  and  pro¬ 
vide  sustaining  engineering  would  be  at  least  $762  million  through 
September  1989.  As  of  March  1991,  we  estimated  that  costs  have 
increased  to  at  least  $906  million  through  September  1991. 2  However, 
the  Program  Office  was  not  able  to  determine  how  much  of  this  increase 
was  due  to  providing  additional  system  capabilities  and  how  much  to 
correcting  system  deficiencies.  Further,  costs  will  continue  to  increase 
because  ccs  will  not  be  completed  until  July  1993.  An  Air  Force  official 
said  a  follow-on  contract  was  awarded  in  September  1991  to  continue 
efforts  to  maintain  and  complete  ccs  (dod  did  not  provide  the  details  in 
time  to  include  in  our  report). 


Further,  because  ccs  is  not  completely  operational  and  some  satellite 
operations  still  depend  on  the  old  cds  system,  the  Air  Force  must  con¬ 
tinue  to  maintain  CDS.  In  1988  the  Air  Force  estimated  that  the  cost  to 
operate  and  maintain  cds  was  $30  million  annually.  Although  Air  Force 
officials  did  not  provide  a  revised  estimate  for  these  costs  during  our 
current  review,  they  noted  that  because  parts  of  cds  are  planned  to  be 
deactivated  by  the  end  of  1991,  CDS  maintenance  costs  should  fall. 


'GAO/IMTEC-89-56,  Aug  8.  1989. 

- 1  kith  estimated  costs  are  based  on  Air  Force  figures  that  include  $458  million  for  the  original  devel¬ 
opment  contract,  as  well  as  the  costs  for  the  sustaining  engineering  contract.  The  estimates  include 
operations  and  maintenance  costs  and  do  not  include  costs  for  fixing  some  critical  deficiencies  for 
which  no  estimate  was  available. 
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CCS  Cannot  Perform 
Key  Command  and 
Control  Operations 


Air  Force  measurements  of  ccs  performance  include  the  percent  of  satel¬ 
lite  contacts  that  are  deemed  successful.  The  Air  Force  requires  a 
95  percent  contact  success  rate  for  ccs.  According  to  the  Program  Office, 
ccs  has  achieved  over  a  95  percent  average  contact  success  rate — up 
from  the  93  percent  we  reported  in  1989.  Further,  ccs  now  supports 
ICO  percent  of  the  total  work-load  that  existed  about  3  years  ago  and  it 
has  supported  every  satellite  launch  in  the  past  year.  In  addition,  the 
percentage  of  contacts  ccs  makes  has  increased  since  we  reported  in 
1 989.  ccs  made  between  90  and  98  percent  of  the  attempted  afscn  con¬ 
tacts  during  the  5  months  ending  April  1991. 3  The  Air  Force  relies  on  CDS 
to  make  the  rest  of  the  contacts. 


Using  this  measure,  the  number  of  successful  contacts  made,  ccs’  per¬ 
formance  has  improved.  However,  the  Air  Force  also  requires  that  most 
mccs  be  able  to  support  at  least  five  simultaneous  satellite  contacts.4 
Using  this  measurement,  ccs  still  comes  up  short,  ccs  cannot  make  the 
required  number  of  simultaneous  contacts  under  all  expected  work  load 
scenarios.5  On  the  basis  of  Air  Force  officials’  comments  and  documenta¬ 
tion  they  provided,  we  found  that  only  two  of  the  eight  mccs  we 
reviewed  can  make  the  required  simultaneous  contacts  under  expected 
peak  work  loads.6  In  fact,  the  operators  sometimes  cut  down  on  the 
number  and  length  of  satellite  contacts  to  make  sure  they  do  not  exceed 
the  system’s  capacity.  For  example,  during  some  satellite  launches  oper¬ 
ators  delay  other  contacts  until  they  feel  comfortable  that  the  work  load 
can  be  processed. 

In  addition  to  the  problem  of  making  five  simultaneous  contacts,  ccs  has 
other  serious  problems — some  affecting  all  satellites  and  others 
affecting  only  a  specific  mcc  or  satellite.  The  Air  Force  calls  these 
problems  critical  deficiencies  because  they  must  be  fixed  before  CDS  can 
be  deactivated  and  ccs  turned  over  to  Space  Command.  Examples  of  sys¬ 
temwide  critical  deficiencies  include  data  base  and  file  contention,7  an 


:iBased  on  weekly  satellite  contact  performance  data  from  December  1990  through  April  1991 . 

4  All  MCCs,  except  two,  are  required  to  support  at  least  five  simultaneous  satellite  contacts.  The  other 
MCCs  are  required  to  support  less  than  five  simultaneous  contacts. 

5These  scenarios  should  include  the  conditions  (e.g.,  the  number  and  types  of  contacts,  the  number 
and  frequency  of  display  updates,  and  the  number  of  commands  per  contact)  that  place  the  expected 
peak  (worst  case)  demand  on  system  resources. 

•’Data  was  available  for  only  eight  MCCs. 

7Contention  occurs  when  programs  that  share  common  resources  (e.g.,  disk  files,  processor)  need  to 
use  the  same  resource  simultaneously.  This  creates  delays  since  one  program  has  to  wait  until  the 
other  is  finished. 
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inability  to  effectively  create  and  delete  files  in  real  time,  and  long 
recovery  times  from  system  failures.  These  kinds  of  problems  could 
damage  critical  files  or  data  bases  needed  to  contact  satellites,  or  delay 
contact  with  or  lose  satellites.  Critical  deficiencies  that  affect  a  specific 
mcc  or  satellite,  such  as  an  inability  to  send  satellite  commands  or  con¬ 
tact  several  satellites  simultaneously,  can  cause  missed  satellite  con¬ 
tacts,  delays  in  maneuvering  and  monitoring  satellites,  delays  in 
correcting  anomalies,  or  even  the  loss  of  a  satellite. 

According  to  Air  Force  officials,  there  are  two  reasons  for  ccs’  problems: 
ccs  software  is  incomplete  and  inefficient,  with  some  functions  missing 
and  others  not  working  correctly;  and  ccs  hardware  capacity  is  insuffi¬ 
cient  to  perform  some  functions.  Air  Force  officials  attributed  ccs’ 
incomplete  software  and  insufficient  capacity,  at  least  in  part,  to  new 
requirements.  Program  Office  officials  said  that  new  requirements  are 
urgent  and  have  to  be  implemented;  they  cannot  be  delayed  until  ccs  is 
turned  over  to  Space  Command. 

The  Program  Office  is  taking  positive  steps  to  correct  ccs’  critical  defi¬ 
ciencies.  First,  it  is  trying  to  fix  software  problems  that  are  critical  to 
command  and  control  and  mission  planning.  Second,  it  is  adding  new 
functions  in  software  releases.  The  fixes  and  changes  are  being  made  in 
software  modifications  that  are  released  approximately  every  6  months. 
Third,  it  is  adding  equipment  to  three  mccs  to  meet  capacity  and  per¬ 
formance  requirements. 


Critical  Deficiencies 
Are  Being  Resolved 
But  Hundreds  Remain 


The  Air  Force  has  corrected  several  hundred  critical  deficiencies  since 
our  last  report.  However,  delays  have  occurred  and  the  current  schedule 
could  slip  again  because  some  deficiencies  have  taken  more  time  to  cor¬ 
rect  than  expected,  correction  of  others  has  been  deferred  as  higher  pri¬ 
ority  problems  arose,  and  new  ones  continue  to  be  identified. 


Eight  hundred  and  ninety-four  critical  deficiencies  existed  in  April  1989, 
when  we  last  reported  on  this  program.8  By  December  1990,  this  number 
had  been  reduced  to  321,  including  some  that  existed  in  April  1989  as 
well  as  some  new  ones  that  developed.  A  Program  Office  official  stated 
that  the  Program  Office  plans  to  fix  all  problems,  as  well  as  any  new 
ones  that  may  arise,  by  completion  of  the  ccs  upgrade  in  1993. 


8GAO/lMTEC-89-56,  Aur.  8. 1989. 
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However,  correcting  ccs’  deficiencies  has  taken  more  time  than  the  Air 
Force  expected.  For  example,  in  February  1990,  Program  Office  officials 
said  they  planned  to  correct  all  known  critical  deficiencies  (569  at  that 
time)  by  September  1991.  Subsequently,  that  date  was  pushed  back 
twice,  first  to  March  1992  and  then  to  September  1992.  Given  that  the 
Air  Force  did  not  meet  its  earlier  two  schedules,  this  latest  estimate  may 
still  not  be  realistic. 

The  former  Program  Manager  for  ccs  said  that,  in  general,  deficiencies 
miss  their  original  correction  date  because  past  attempts  to  fix  the  defi¬ 
ciencies  failed,  or  the  deficiencies’  correction  dates  were  delayed 
because  other,  more  critical  problems  arose  that  had  to  be  resolved  first. 

The  Air  Force  also  continues  to  find  new  critical  deficiencies  in  each  new 
software  release.  For  example,  175  new  critical  deficiencies  were  identi¬ 
fied  between  March  1989  and  October  1990.  According  to  the  Air  Force, 
each  new  deficiency  is  evaluated  and  scheduled  to  be  corrected  as  it 
arises.  These  deficiencies  must  be  corrected  before  CCS  can  be  fully  oper¬ 
ational.  However,  as  of  December  1990,  some  of  these  had  not  even  been 
scheduled  to  be  fixed. 

These  deficiencies  must  be  corrected  before  ccs  can  function  as 
intended.  Furthermore,  we  believe  that  the  Air  Force  has  four  additional 
problems  that  jeopardize  the  likelihood  of  ccs’  success:  inadequately 
defined  work -load  scenarios;  inadequate  testing;  an  inadequate  capacity 
and  performance  management  program;  and  poor  software  documenta¬ 
tion.  These  problems  are  addressed  in  chapter  3. 
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We  are  concerned  that  the  Air  Force  has  not:  adequately  defined 
expected  peak  work-load  requirements  for  CCS;  adequately  tested  ccs; 
set  up  an  adequate  capacity  and  performance  management  program; 
and  obtained  adequate  software  documentation.  These  proolems  raise 
serious  doubts,  not  only  about  the  Air  Force’s  ability  to  complete  ccs  by 
July  1993,  but  also  about  ccs’  capability  to  meet  its  mission 
requirements. 


CCS  Requirements  Are 
Not  Adequately 
Defined 


The  ccs  contract  requires  compliance  with  Military  Standard  483A, 
“Configuration  Management  Practices  for  Systems,  Equipment,  Muni¬ 
tions,  and  Computer  Programs,”  and  Military  Standard  490A,  “Specifi¬ 
cation  Practices.”  These  military  standards  call  for  detailed 
requirements  to  be  prepared  when  acquiring  or  modifying  computer  sys¬ 
tems,  including  peak  processing  work  loads.  The  expected  peak  work 
loads  are  critical  to  assuring  that  the  system  will  function  effectively 
under  maximum  expected  stress.  These  work  loads  are  used  to  help  size 
the  system  for  current  and  future  needs,  to  develop  tests  to  stress  the 
system,  and  as  input  for  capacity  and  performance  management 
activities. 


In  the  case  of  ccs,  these  requirements  are  defined  by  work-load  scena¬ 
rios  that  describe  such  factors  as:  the  mix  of  satellites  and  the  types  of 
contacts  that  must  be  controlled  simultaneously  during  a  certain  time 
period,  the  number  of  terminals  the  system  must  support,  and  the 
amount  of  data  that  must  be  processed.  These  scenarios  should  include 
those  that  place  the  greatest  expected  demand  on  system  resources  (i.e., 
the  expected  peak  work  load  or  worst-case  scenario).  For  example,  the 
worst-case  scenario  for  one  mcc  involves  simultaneously  performing: 
command  and  telemetry  contacts  for  three  satellites,  a  mission  data  con¬ 
tact  for  a  fourth  satellite,  and  a  state  of  health  contact  for  a  fifth 
satellite. 

Because  mccs  are  set  up  to  support  specific  types  and  numbers  of  satel¬ 
lites,  each  mcc  has  its  own  set  of  work-load  scenarios.  The  Air  Force  last 
updated  its  expected  peak  work  loads  for  the  mccs  in  1987.  However, 
since  then,  the  number  of  satellites  has  increased  by  36  percent  and, 
according  to  Program  Office  officials,  the  mccs  are  handling  different 
types  and  numbers  of  satellites.  Therefore,  the  1987  expected  peak 
work  loads  are  obsolete.  For  example,  the  mcc  that  handles  the  Space 
Shuttle  and  another  space  vehicle  program  now  also  supports  a  third 
program.  However,  its  work-load  requirement  is  still  based  only  on  the 
first  two  programs;  it  has  not  been  updated  to  include  the  work  load 


Page  18 


GAO  IMTEC-92-3  Satellite  Control  System  Upgrade 


Chapter  3 

Inadequate  Requirements  Definition,  Testing, 
Capacity  Management,  and  Software 
Documentation  Puts  CCS  at  Risk 


generated  by  the  third  program.  Without  updated  requirements,  the  Air 
Force  cannot  assure  that  the  system  is  properly  sized  for  current  and 
future  needs,  it  cannot  develop  tests  to  adequately  stress  the  system, 
and  it  cannot  reliably  assess  the  impact  of  continual  ccs  changes  on  mis¬ 
sion  effectiveness. 

Space  Command  officials  said  that  because  of  other  priorities — they  are 
concentrating  on  trying  to  identify  ccs  problems  and  working  with  the 
Program  Office  and  mccs  to  correct  them — they  have  not  updated  the 
worst-case  work  loads.  They  said  that  they  are  working  to  define  the 
criteria  for  the  mccs’  operational  tests  and  evaluations  (see  next  sec¬ 
tion),  which  will  include  testing  worst-case  scenarios. 


Testing  Inadequate 


Expected  peak  work  loads  are  not  only  critical  to  developing  a  system, 
but  they  are  essential  for  testing  a  system  and  its  modifications  to  deter¬ 
mine  if  the  system  can  do  its  job.  If  testing  is  inadequate,  design  flaws 
and  errors  may  not  be  found  until  the  system  is  put  in  operation.  At  this 
late  stage,  problems  are  more  difficult  and  expensive  to  correct  than  if 
they  had  been  discovered  and  fixed  earlier.  Moreover,  finding  a  critical 
problem  while  ccs  is  helping  launch  or  track  a  satellite  could  jeopardize 
that  satellite. 

Until  the  mccs  have  updated  work-load  requirements,  the  Air  Force 
cannot  (1)  operationally  test  the  current  ccs  configuration  to  ensure  that 
the  more  taxing  satellite  contact  scenarios  can  be  met  before  the  mccs 
are  turned  over  to  Space  Command,  and  (2)  stress  test  each  software 
release  in  a  laboratory  setting  before  it  is  used  in  the  mccs.  Further,  the 
testing  that  is  being  done  on  new  software  releases  is  not  thorough 
enough  to  uncover  critical  problems  before  ccs  is  used  in  an  operational 
environment  in  the  mccs. 


Air  Force  Operational 
Testing  and  Stress  Testing 
of  CCS  Not  Based  on 
Updated  Work-Load 
Requirements 


The  Air  Force  Operational  Test  and  Evaluation  Center  (apotec)  is 
required  to  test  all  the  mccs  at  Falcon  Air  Force  Base  before  they  are 
turned  over  to  Space  Command.  This  operational  testing  and  evaluation 
is  to  determine  the  mccs’  operational  effectiveness  and  suitability,  and  is 
a  critical  milestone  for  ccs’  turnover.  It  involves  (1)  testing  the  com¬ 
puter’s  software  and  hardware,  (2)  checking  procedures,  and  (3)  certi¬ 
fying  personnel,  apotec  and  Space  Command  officials  said  one  key  test 
is  determining  if  the  system  can  perform  the  required  number  of  simul¬ 
taneous  contacts  under  expected  peak  work-load  conditions.  However, 
because  the  Air  Force  has  not  updated  the  mccs’  expected  peak  work- 
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load  requirements,  apotec  cannot  conduct  effective  tests  and 
evaluations. 

Further,  major  ccs  software  releases  are  currently  being  issued  approxi¬ 
mately  twice  a  year.  These  new  releases  are  intended  to  correct  existing 
software  problems  and  add  new  functions.  However,  without  a  current 
worst-case  work-load  scenario,  the  software  cannot  be  stress  tested  to 
see  how  it  performs  under  maximum  work-load  conditions. 


Software  Testing  Not 
Identifying  All  Critical 
Problems 


ccs  software  is  not  being  adequately  tested  before  it  is  put  into  an  opera¬ 
tional  environment  in  the  mccs.  Critical  deficiencies  continue  to  be  found 
after  new  and  revised  software  is  released  for  use.  This  is  due,  at  least 
in  part,  to  the  fact  that  the  test  center’s  Software  Development  Testing 
Laboratory,  which  is  responsible  for  testing  ccs  software  releases,  is  not 
equipped  to  simulate  the  high  stress  work  loads  that  mccs  must  support. 

All  but  two  mccs  are  required  to  perform  at  least  five  simultaneous  con¬ 
tacts.  However,  according  to  Program  Office  officials,  the  lab  can  only 
simulate  the  work  load  generated  by  three  simultaneous  satellite  con¬ 
tacts.  As  a  result,  testing  at  the  lab  does  not  provide  information  on 
whether  the  system  can  perform  more  than  three  simultaneous  contacts. 
The  Air  Force  has  no  way  of  reliably  assessing  how  the  system  will  per¬ 
form  under  more  stringent  conditions  until  it  installs  the  new  software 
in  the  mccs  and  uses  ccs  to  command  and  control  the  satellites. 

Because  the  software  is  not  fully  tested,  most  critical  deficiencies  are 
not  found  until  the  mccs  use  the  software.  Between  March  1989  and 
March  1990,  three  major  software  releases  were  tested  at  the  lab;  no 
new  critical  deficiencies  were  found.  However,  after  these  releases  were 
used  in  the  mccs,  operators  found  about  175  new  critical  deficiencies. 

For  example,  one  ccs  function  does  not  always  accurately  predict  satel¬ 
lite  positioning,  which  could  result  in  an  incorrect  satellite  maneuver. 
Program  Office  officials  said  they  believe  the  majority  of  critical  defi¬ 
ciencies  were  not  identified  because  of  the  inability  to  test  the  software 
in  scenarios  involving  more  than  three  simultaneous  contacts. 

Both  Program  Office  and  sustaining  engineering  contractor  officials 
believe  the  limited  test  capabilities  (i.e.,  no  more  than  three  simulta¬ 
neous  contacts)  hurts  their  ability  to  adequately  test  new  software 
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releases.1  Program  Office  officials  believe  testing  is  important  and  have 
requested,  but  not  received,  funds  to  improve  the  test  facilities. 


Computer  Capacity 
and  Performance 
Management  Not 
Adequate 


Capacity  and  performance  management  is  the  process  by  which  the 
components  of  the  computerized  system  are  configured,  used,  and  main¬ 
tained  to  assure  that  the  current  and  projected  work  loads  are  processed 
effectively,  efficiently,  and  economically.  Federal  regulations  and  guid¬ 
ance  recognize  the  importance  of  conducting  capacity  management 
activities  to  plan,  acquire,  and  use  computer  resources.2  In  addition,  the 
General  Services  Administration’s  Federal  Systems  Integration  and  Man¬ 
agement  Center  considers  capacity  management  (including  performance 
management)  to  be  an  important  tool  in  effectively  using  information 
processing  systems.  Such  activities  are  important  because  they  provide 
agencies  with  information  about  current  system  utilization  and  addi¬ 
tional  capacity  that  may  be  required  to  meet  future  needs. 

Failure  to  monitor  capacity  use  and  anticipate  capacity  limitations  can 
have  potentially  damaging  consequences.  For  example,  the  Federal  Avi¬ 
ation  Administration  uses  computers  to  handle  air  traffic  control,  but 
due  to  capacity  limitations,  data  describing  air  traffic  began  disap¬ 
pearing  from  controllers’  computer  terminals.  This  occurred,  in  part, 
because  the  agency  did  not  know  how  much  computer  capacity  was 
being  used  during  normal  operations.3 

To  manage  capacity  effectively,  managers  must  regularly  monitor 
system  performance  and  capacity  utilization.  They  must  make  reasoned 
predictions  about  future  work  loads;  determine  how  proposed  and 
actual  changes  to  the  system  will  affect  system  performance;  and  make 
recommendations  concerning  system  configuration  and  operation  to 
assure  good  service  to  users  now  and  in  the  future.  Performance  data, 
gathered  by  system  facilities,  hardware  monitors,  and  software 
monitors,  and  the  use  of  effective  analytic  modeling  tools  and  tech¬ 
niques  are  essential  in  managing  capacity  effectively. 

The  Air  Force  has  not  set  up  an  effective  capacity  and  performance 
management  program,  even  though  it  acknowledges  that  ccs  has 


'CCS-  sustaining  engineering  contractor  analyzes  the  system  and  its  interfaces;  designs  and  imple¬ 
ments  hardware,  firmware,  and  software  modifications;  and  maintains  the  software. 

2E.g.,  the  Federal  Information  Resources  Management  Regulation. 

3Air  Traffic  Control:  Computer  Capacity  Shortfalls  May  Impair  Flight  Safety  (GAO/IMTEC-89-63, 
July  6,  1989). 
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capacity  problems  in  three  mccs.  The  Program  Office  relies  on  ccs  satel¬ 
lite  contact  success  data,  the  computer  operators’  perceptions  of  the 
system’s  limitations,  and  infrequent  ad  hoc  analyses  of  computer 
capacity  and  performance  to  manage  computer  resources.  However, 
these  only  provide  indications  of  system  performance — they  do  not 
measure  actual  use  or  reliably  assess  performance  on  a  continuous  basis. 

First,  the  analysis  that  was  done  using  the  information  collected  is  lim¬ 
ited.  For  instance,  the  Program  Office  collects  data  on  the  number  of 
successful  and  failed  contacts  using  ccs,  but  does  not  perform  detailed 
analyses  to  pinpoint  the  causes  of  failed  contacts.  Failed  contacts  are 
only  attributed  to  affected  areas,  such  as  ccs  computer  hardware,  ccs 
software,  or  other  network  resources.  The  data  do  not  provide  enough 
information  to  determine  what  actually  caused  the  problem. 

Second,  relying  on  the  system  operators’  perceptions  to  manage  capacity 
is  also  not  an  effective  management  tool.  Program  Office  officials  said 
that  operators  manage  satellite  command,  control,  and  mission  data 
processing  work  loads  to  stay  within  what  they  believe  are  the  system’s 
capacity  and  performance  parameters.  Operators  come  up  with  ways  to 
work  around  system  problems  that  are  known  to  occur.  Basically,  the 
operators  cut  down  on  the  number  or  length  of  satellite  contacts  they 
make  at  certain  times  to  make  sure  that  computer  use  stays  within  the 
perceived  capacity  and  performance  limits.  However,  working  this  way 
does  not  tell  management  how  well  ccs  is  performing  its  mission,  what 
capacity  and  performance  limitations  actually  are,  or  identify  what  type 
and  when  improvements  in  computer  resources  are  needed  to  increase 
capacity. 

Third,  conducting  infrequent  ad  hoc  computer  capacity  and  perform¬ 
ance  analyses  is  inadequate.  According  to  Program  Office  officials,  these 
analyses  are  narrowly  focused — they  only  address  specific  questions 
regarding  the  mccs’  ability  to  perform  selected  work-load  scenarios  at 
isolated  points  in  time.  The  analytical  results  are  not  detailed  enough  to 
assess  system  performance  and  capacity,  and  they  cannot  trace 
problems  to  particular  processing  events.  Further,  these  analyses  do  not 
always  adequately  describe  the  methods  used  or  the  specific  work-load 
scenarios,  and  they  are  not  done  on  a  regular  basis.  Therefore,  they 
cannot  be  used  to  show  trends  in  capacity  use  and  performance. 

Officials  from  Space  Command’s  Second  Space  Wing  agreed  that  the 
indicators  currently  being  used  were  inadequate  to  manage  ccs  capacity. 
The  former  ccs  Program  Manager  agreed  that  two  of  the  indicators — the 
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contact  success  data  and  ad  hoc  analyses — were  inadequate  to  assess 
capacity,  but  he  stated  that  the  operators’  knowledge  of  how  the  system 
is  working  is  a  sufficient  tool  to  manage  capacity.  Program  Office  offi¬ 
cials  added  that  despite  the  problems  with  ad  hoc  analyses,  they  will 
continue  to  use  them  because  they  believe  conducting  comprehensive 
computer  capacity  and  performance  analyses  are  too  expensive.  Regard¬ 
less  of  the  cost,  Program  Office  officials  also  claimed  that  the  mccs’  com¬ 
puters  do  not  have  the  capacity  to  do  capacity  and  performance 
analyses  on  a  regular  basis  since  such  analyses  would  overload  the 
system.  However,  the  Program  Office  did  not  provide  any  documenta¬ 
tion  supporting  their  claims  that  their  ad  hoc  analyses  were  sufficient, 
or  that  systematic  capacity  utilization  and  performance  measurement 
would  be  too  expensive  and  resource-intensive. 

The  Program  Office  is  not  planning  to  improve  capacity  and  perform¬ 
ance  analyses  in  the  future,  ccs’  sustaining  engineering  contractor  asked 
the  Program  Office  in  February  1990  to  allow  it  to  measure  ccs’  per¬ 
formance,  including  computer  capacity  and  use,  within  the  mccs.  The 
contractor  believed  this  would  help  in  understanding  the  mccs’  capabili¬ 
ties,  fixing  existing  problems,  and  preventing  future  ones.  The  Program 
Office,  however,  did  not  approve  this  request,  citing  limited  funds  and 
the  need  for  the  contractor  to  first  solve  ccs’  software  and  hardware 
problems. 

On  the  other  hand,  the  Second  Space  Wing  recognizes  that  it  will  need 
better  sources  of  information  when  it  assumes  responsibility  for  its  por¬ 
tions  of  ccs  software  in  1993.  In  March  1990,  the  Wing  reorganized  its 
computer  performance  branch  to  place  more  emphasis  on  performance 
and  capacity  planning  and  analysis.  This  reorganization  is  a  construc¬ 
tive  step  towards  assessing  performance  and  capacity. 


A  fourth  problem  that  also  affects  the  Air  Force’s  ability  to  upgrade, 
test,  modify,  and  maintain  ccs  is  poor  software  documentation.  Federal 
Information  Processing  Standards  (Publication  38)  recognize  that  good 
computer  system  software  documentation  is  needed  for  effectively  man¬ 
aging  computer  resources  and  expediting  software  changes. 

Software  documentation  should  be  complete,  up-to-date,  and  well  organ¬ 
ized.  It  should  describe:  1)  requirements;  2)  design  specifications; 

3)  users,  operations,  and  program  maintenance  procedures;  4)  test  plans; 
and  5)  test  results  and  findings.  Without  adequate  documentation, 
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software  maintenance  takes  more  time  and  effort  and  there  is  less  assur¬ 
ance  that  software  modifications  will  function  as  required. 

ccs  system  software  documentation  is  incomplete,  out-of-date,  and  diffi¬ 
cult  to  use.  According  to  apotec  officials,  inadequate  documentation  rep¬ 
resents  a  significant  risk  to  satellite  control  operations.  Specifically,  it 
impairs  the  Air  Force’s  ability  to  identify  and  correct  problems  in  a 
timely  and  cost-effective  way. 

In  preparation  for  Space  Command  assuming  responsibility  for  ccs 
software,  apotec  is  evaluating  ccs’  software  documentation.  As  of  April 
1991,  it  had  evaluated  12  of  ccs’  28  software  configuration  items.4  It 
found  that  most  of  the  documentation  were  less  than  acceptable — some 
were  missing,  some  were  incomplete,  and  some  were  out-of-date,  apotec 
concluded  that  the  existing  documentation  will  make  the  system  diffi¬ 
cult  to  maintain  and  that  improvements  are  needed.  Table  3.1  shows 
apotec’s  evaluation  results. 


able  3.T.  AFOTEC’s  Evaluation  of 
oftware  Documentation  for  12  CCS 

Level  of  Documentation  Quality 

oftware  Configuration  Items  at  the 
Onsolidated  Space  Operations  Center 

Type  of  Software  Items 

Unacceptable 

Unacceptable 
to  Marginal 

Marginal 

Acceptable 

Common  User3 

4 

1 

• 

• 

Mission  Uniqueb 

1 

2 

2 

2 

Total 

5 

3 

2 

2 

aCommon  user  software  items  are  used  in  every  MCC  There  are  19  common  user  software  configura¬ 
tion  items  in  use  at  the  Operations  Center. 


^Mission  unique  software  items  are  unique  to  particular  MCCs  and  support  a  specific  satellite  program 
There  are  nine  mission  unique  software  configuration  items  in  use  at  the  Operations  Center 


Based  on  afctec’s  evaluations,  software  documentation  that  is  unaccept¬ 
able  or  unacceptable-to-marginal  means  that  the  software  will  be  diffi¬ 
cult  to  maintain  because  of  either  poor  or  missing  documentation,  apotec 
reported  that  this  type  of  documentation  will  generally  need  many 
changes  before  it  can  be  used  as  an  effective  maintenance  tool.  A  mar¬ 
ginal  assessment  by  apctec  means  that  while  the  documentation  is 
useful  as  a  maintenance  tool,  the  documentation  was  incomplete.  As  a 
result,  it  will  take  more  time,  effort,  and  resources  to  maintain  the 
software  (for  example,  to  find  problems  and  fix  them). 


4P  ir  the  purpose  of  this  report,  a  software  configuration  item  ean  be  either  a  single  software  modulo 
or  an  aggregation  of  software  modules  that  perform  a  specific  function  or  process 
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Poor  documentation  also  makes  it  more  difficult  for  programmers  to 
understand  the  code  and  its  logic.  Therefore,  they  may  make  modifica¬ 
tions  that  cause  errors  or  that  run  inefficiently,  i.e.,  that  unnecessarily 
consume  capacity  and  potentially  degrade  system  performance.  The 
former  ccs  Program  Manager  stated  that  Program  Office  officials 
believe  the  combination  of  inadequate  software  documentation  and 
ccs’  inefficiently  written  software  modifications  has  resulted  in  ineffi¬ 
cient  use  of  computer  resources.  The  former  ccs  Program  Manager  also 
stated  that  inefficient  software  was  one  reason  why  most  mccs  cannot 
perform  the  required  maximum  number  of  simultaneous  satellite  con¬ 
tacts  during  expected  peak  work  loads.  However,  according  to  this  offi¬ 
cial,  no  evidence  is  available  to  support  this  point. 

Program  Office  and  afotec  officials  attributed  inadequate  software  doc¬ 
umentation  to:  (1)  poor  management  control  over  the  initial  ccs  develop¬ 
ment,  and  (2)  an  overriding  interest  in  getting  an  operational  system 
within  budget  and  on  schedule.  While  the  Program  Office  believes  the 
documentation  is  adequate  to  maintain  the  software,  apoi  EC’s  findings 
show  otherwise.  We  believe  that  adequate  software  documentation  is 
essential  for  maintaining  and  modifying  a  system  in  a  cost-effective  and 
efficient  manner. 

dod  audits,  our  reports,  and  other  studies  have  shown  repeatedly  that 
poor  software  documentation  increases  maintenance  costs,  which  com¬ 
prise  the  largest  percentage  of  a  system’s  life  cycle  costs.  According  to 
Barry  W.  Boehm,5  estimates  of  the  magnitude  of  software  maintenance 
range  from  slightly  over  50  to  75  percent  of  overall  software  life  cycle 
costs,  dod’s  Inspector  General  reported  in  1988  that  the  computer 
industry  estimates  that  about  60  percent  of  programming  resources  are 
dedicated  to  the  modification  and  maintenance  of  existing  software.6  It 
is  widely  recognized  that  shortchanging  documentation  is  a  poor 
strategy  both  economically  and  operationally.  We  have  reported  that 
without  good  documentation,  software  is  difficult  to  understand  and 
maintain.7  Organizations  have,  in  fact,  chosen  to  redesign  and  rebuild 
systems  because  understanding  and  modifying  the  poorly  documented 
existing  system  was  so  difficult. 


r,Barry  W.  Boehm.  Software  Engineering  Economics,  (Prentice-Hall,  Inc.,  1981). 

'’Summary  Report  on  the  Defense-wide  Audit  of  Support  for  Tactical  Software.  (Department  of 
Defense.  Office  of  the  Inspector  General,  April  1988) 

"Federal  Agencies'  Maintenance  Of  Computer  Programs:  Expensive  and  Undermanaged  (GAO- 
AFMt>-81-25,  Feb.  2(i.  1981). 
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Conclusions 


The  Air  Force  is  working  to  complete  ccs’  upgrade  by  fixing  software 
problems,  adding  new  functions  to  ccs,  and  adding  equipment  to  specific 
mccs.  These  efforts  have  fixed  many  of  ccs’  problems  and  ccs  is  now 
used  to  make  most  satellite  contacts.  However,  ccs  costs  continue  to 
increase,  and  new  software  problems  continue  to  be  identified.  The  Air 
Force  may  not  be  able  to  correct  these  problems  and  ccs  may  not  have 
all  of  the  functions,  capacity,  and  perform anc  ■  needed  by  1993  because 
the  Air  Force  has  not:  adequa.  My  defined  work-load  requirements  for 
ccs;  adequately  tested  ccs;  se  up  an  effective  capacity  and  performance 
management  program:  and  obtained  adequate  software  documentation. 

Without  up-to-date  work-load  requirements,  the  Air  Force  cannot  assure 
tl  at  the  system  is  properly  sized  for  current  and  future  needs,  it  cannot 
develop  tests  to  adequately  stress  the  system,  and  it  cannot  reliably 
assess  the  impact  of  ccs  changes  on  mission  effectiveness.  Further,  until 
the  work  loads  are  updated,  the  Air  Force  cannot  determine  if  ccs  can 
meet  the  mccs’  operational  requirements,  nor  can  it  adequately  stress 
test  ccs  software  releases.  In  addition,  since  the  Air  Force  does  not  have 
n  effective  capacity  and  performance  management  program,  it  does  not 
know  how  well  CCS  is  performing  and  it  cannot  determine  how  much 
more  capacity,  if  any,  is  needed  to  meet  the  ever-changing  future 
requirements.  Finally,  unless  the  Air  Force  corrects  deficiencies  in  ccs’ 
software  documentation,  it  will  be  increasingly  difficult  to  maintain  and 
enhance  ccs  as  requirements  continue  to  change  and  evolve,  and  the 
costs  to  maintain  ccs  are  likely  to  continue  to  increase. 

Unless  all  of  these  problems  are  resolved  soon,  the  Air  Force  is  assuming 
a  significant  risk  that  ccs  may  not  have  the  functions,  capacity,  and  per¬ 
formance  needed  to  command,  control,  and  plan  satellite  missions.  Fur¬ 
ther,  if  ccs’  upgrade  misses  its  deadline,  turnover  of  the  mccs  to  Space 
Command  may  be  delayed,  and  the  Air  Force  would  be  forced  to  con¬ 
tinue  spending  up  to  $30  million  annually  to  maintain  an  outdated 
system. 


Recommendations 


To  increase  the  Air  Force’s  ability  to  plan,  measure,  and  control  the  com¬ 
puter  resources  required  to  meet  current  and  future  satellite  command 
and  control  and  mission  planning  processing  requirements,  we  recom¬ 
mend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Air  Force 
to  ensure  that: 
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CCS  work-load  requirements  (specifically,  the  peak  work-load  require¬ 
ments  for  each  mcc’s  computer  system)  are  immediately  updated  and 
kept  current; 

the  updated  work- load  requirements  are  used  to  (1)  operationally  test 
ccs  at  the  mccs  before  the  mccs  are  turned  over  to  Space  Command,  and 
(2)  stress  test  each  ccs  software  release  before  it  is  used  in  the  mccs; 
a  comprehensive  ccs  capacity  and  performance  management  program  is 
immediately  established;  and 

ccs’s  software  documentation  deficiencies  are  corrected  before  the 
system  is  turned  over  to  Space  Command,  and  documentation  is  ade¬ 
quately  prepared  and  maintained  in  the  future. 


In  commenting  on  a  draft  of  this  report,  dod  partially  concurred  with  all 
the  findings  but  one.  dod  believes  that  our  updated  estimates  for  total 
system  costs  incorrectly  combined  acquisition  costs  (associated  with  the 
completed  development  of  CCS)  and  subsequent  life  cycle  costs.  We 
believe  that  DOD  incorrectly  characterizes  the  cost  information  in  the 
report,  which  clearly  states  that  the  costs  include  those  for  developing 
ccs  and  those  for  the  sustaining  engineering  needed  to  complete  develop¬ 
ment  and  to  satisfy  new  requirements.  Since  the  purpose  of  ccs  was  to 
replace  the  old,  out-of-date  CDS,  we  believe  it  is  appropriate  to  report  all 
costs  until  all  original  requirements  for  ccs  have  been  completed  and  CDS 
is  deactivated.  As  noted  in  the  report,  the  Air  Force  was  not  able  to 
determine  how  much  of  the  sustaining  engineering  costs  were  for 
meeting  original  requirements  and  how  much  were  for  new 
requirements. 

dod  partially  concurred  with  our  finding  that  ccs  cannot  perform  key 
operations.  Its  partial  nonconcurrence  appears  to  center  on  the  Air 
Force’s  position  that,  today,  ccs  is  supporting  virtually  all  scheduled  sat¬ 
ellite  contacts — a  position  that  was  clearly  recognized  in  our  draft 
report.  However,  dod  agreed  that  most  mccs  cannot  make  five  simulta¬ 
neous  contacts  under  all  expected  work-load  scenarios.  We,  therefore, 
continue  to  believe  that  ccs  may  not  meet  the  five  simultaneous  contact 
requirement  at  turnover  in  July  1993,  particularly  as  work  loads  con¬ 
tinue  to  change  and  grow. 

dod  partially  concurred  with  our  finding  that  critical  deficiencies  still 
remain,  and  agreed  with  our  finding  that  correcting  some  critical  defi¬ 
ciencies  has  taken  longer  than  originally  expected.  However,  dod 
objected  to  what  it  characterized  as  our  combining  deficiencies  associ¬ 
ated  with  the  development  phase  and  those  associated  with  new 
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evolving  requirements,  and  it  asserts  that  the  last  six  turnover  critical 
deficiencies  are  scheduled  to  be  corrected  by  June  1992. 

The  321  critical  deficiencies  that  we  identified  were  based,  as  the  report 
clearly  states,  on  the  Air  Force’s  criteria  for  critical  deficiencies — defi¬ 
ciencies  that  must  be  fixed  before  CDS  can  be  deactivated  and  ccs  turned 
over  to  Space  Command.  The  additional  information  the  Air  Force  sup¬ 
plied  to  support  its  position  was  inconsistent  with  the  information  previ¬ 
ously  provided  and  the  Air  Force  did  not  provide  explanations  for  these 
inconsistencies.  Further,  the  Air  Force  did  not  provide  an  explanation  of 
the  disposition  of  the  321  critical  deficiencies.  Therefore,  we  have  not 
revised  the  report  to  present  this  new  information. 

dod  partially  concurred  with  our  conclusions  and  recommendation  on 
updating  and  keeping  work-load  requirements  current,  dod  stated  that 
the  Air  Force  has  defined  work-load  requirements  for  turnover  of  the 
mccs  and  that  it  will  define  and  continue  to  evaluate  evolving  peak 
work-load  requirements  for  all  mccs  as  new  missions  are  added.  We 
believe  that  the  Air  Force’s  actions  respond  to  our  concerns.  We  were 
unable  to  evaluate  the  adequacy  of  the  updated  requirements  since  this 
information  was  not  supplied  to  us  when  dod  provided  its  comments. 

dod  partially  concurred  with  our  conclusions  and  recommendations  on 
ccs  testing,  dod  stated  that  the  Air  Force  is  identifying  the  tests  neces¬ 
sary  to  validate  ccs  for  operational  turnover  to  Space  Command  consis¬ 
tent  with  satellite  program  requirements,  dod  noted  that  while  these 
requirements  do  not  necessarily  represent  current  peak  work  loads,  the 
Air  Force  will  ensure  that  modifications  to  ccs  are  made  as  necessary  to 
meet  evolving  work-load  requirements,  dod’s  response  recognizes  the 
need  to  ensure  that  the  updated  work-load  requirements  are  used  to 
operationally  test  ccs  at  mccs  before  they  are  turned  over  to  Space 
Command. 

dod  agreed  that  the  Air  Force’s  current  software  test  tools  are  not  ade¬ 
quate  to  stress  test  ccs  at  peak  work  loads  before  it  is  used  in  the  mccs. 
However,  dod  was  silent  on  requiring  the  Air  Force  to  conduct  these 
tests,  dod  stated  the  Air  Force  has  plans  to  upgrade  the  test  environ¬ 
ment,  but  noted  that  these  requirements  will  compete  for  funding  in  the 
budget  cycle.  While  we  recognize  that  dod  and  the  Air  Force  are  under 
funding  constraints,  we  believe  that  dod  should  take  a  stronger  position 
on  requiring  testing  and  ensuring  funds  are  spent  on  upgrading  the  test 
facilities.  Testing  is  extremely  important  to  assuring  that  ccs  can  meet 
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its  mission,  and  correcting  problems  before,  instead  of  after,  implemen¬ 
tation  should  save  funds  over  the  long  run. 

dod  partially  concurred  with  our  conclusions  and  recommendation  on 
ccs  capacity  and  performance  management,  dod  stated  that  the  current 
methods  are  sufficient  and  it  did  not  require  the  Air  Force  to  establish  a 
formal  capacity  and  performance  management  program.  However,  dod 
directed  the  Air  Force  to  assess  its  current  methods  and  determine  how 
to  improve  capacity  and  performance  management.  The  Air  Force’s  cur¬ 
rent  efforts  to  manage  ccs  do  not  provide  the  information  necessary  to 
adequately  manage  system  capacity  and  performance.  We  believe  that 
getting  the  Air  Force  to  assess  its  current  efforts  is  a  step  in  the  right 
direction  and  should  demonstrate  that  it  needs  a  capacity  and  perform¬ 
ance  management  program  that  regularly  monitors  system  performance 
and  capacity  utilization. 

dod  partially  concurred  with  our  conclusions  and  recommendation  on 
software  documentation,  dod  stated  that  apotec  found  that  some  docu¬ 
mentation  is  less  than  acceptable  and  the  Program  Office  has  under¬ 
taken  action  to  upgrade  some  of  the  documentation.  However,  dod 
asserts  that  apotec’s  findings  should  not  be  interpreted  as  indicating 
that  the  documentation  is  unacceptable  for  turnover.  We  disagree. 
apotec  concluded  that  the  unacceptable  documentation  will  make  it  dif¬ 
ficult  to  maintain  the  software  and  that  some  portions  will  need  changes 
before  the  documentation  can  be  used  as  an  effective  maintenance  tool. 
apotec’s  conclusion  makes  it  clear  that  portions  of  the  documentation 
are  unacceptable  for  turnover. 

Further,  dod  cited  the  Air  Force’s  Competition  Advocate  General’s  eval¬ 
uation  of  the  competition  for  the  new  sustaining  engineering  contract  as 
support  for  ccs’  software  documentation  being  adequate  for  maintaining 
the  system.  To  evaluate  the  documentation,  the  Competition  Advocate 
General  discussed  its  adequacy  with  one  potential  bidder.  He  did  not 
evaluate  ccs  software  documentation  nor  did  he  review  apotec’s  find¬ 
ings.  His  conclusion  that  there  were  no  documentation  deficiencies  that 
precluded  fair  competition  is  unrelated  to  the  question  of  whether  the 
documentation  is  an  effective  tool  for  maintaining  the  software. 
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DIRECTOR  OF  DEFENSE  RESEARCH  AND  ENGINEERING 


WASHINGTON,  DC  20301-3010 


Mr.  Ralph  V.  Car lone 
Assistant  Comptroller  General 
Information  Management  and 
Technology  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Car lone: 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  Draft  Report,  "MILITARY  SPACE 
OPERATIONS:  Satellite  Control  System  Improved,  But  Serious 
Problems  Remain,"  dated  September  20,  1991  (GAO  Code  510474,  OSC 
Case  8745) . 

Although  the  Department  partially  concurs  with  most  of  the 
findings  and  all  of  the  recommendations,  the  DoD  non-concurs  wit.ii 
the  finding  entitled  "Command  and  Control  Segment  Costs  Continue 
To  Increase."  The  fundamental  disagreement  concerns  combining 
initial  acquisition  costs  associated  with  the  development  of  tho 
Command  and  Control  Segment  and  normal  life-cycle  costs 
associated  with  its  operation. 

Additional  DoD  comments  to  the  draft  report  findings  are 
enclosed.  Thank  you  for  the  opportunity  to  comment  on  the  draft 
report. 

Sincerely, 


Charles  E.  fljdolph 

By  Direction  of  the  Secretary  of  Defen:' 


Page  31 


GAO/IMTEC-92-3  Satellite  Control  System  Upgrade 


Appendix  II 

Comments  From  the  Department  of  Defense 


See  comment  1 


GAO  DRAFT  REPORT  -  DATED  SEPTEMBER  20,  1991 
(GAO  CODE  510474)  OSD  CASE  8745 

"MILITARY  SPACE  OPERATIONS:  SATELLITE  CONTROL  SYSTEM  IMPROVED, 
BUT  SERIOUS  PROBLEMS  REMAIN" 

DEPARTMENT  OF  DEFENSE  COMMENTS 


FINDINGS 


•  Finding  A:  Command  and  Control  Segment  Costs  Continue  To  Increase,  The  GAO 
reported  that,  since  1980,  the  Air  Force  Systems  Command  has  been  upgrading  its  satellite 
command  and  control  system,  called  the  Command  and  Control  Segment.  The  GAO  found 
that,  since  it  last  reported  on  the  effort  in  1989  (GAO/IMTEC-89-56,  OSD  Case  8083),  the 
Air  Force  has  made  considerable  progress  in  upgrading  the  Segment.  The  GAO  reported,  for 
example,  that  the  system  is  now  handling  a  larger  portion  of  satellite  contacts  and  several 
hundred  critical  design  deficiencies  have  been  fixed.  At  the  same  time,  however,  the  GAO 
reported  that  the  cost  to  complete  the  Segment  continues  to  increase,  requirements  continue 
to  expand,  and  new  system  deficiencies  are  being  found. 

With  regard  to  costs,  the  GAO  noted  that,  based  on  Air  Force  figures  as  of  March  1989,  and 
an  anticipated  June  1991  completion  date,  it  previously  estimated  the  cost  to  develop  the 
Command  and  Control  Segment  and  provide  sustaining  engineering  would  be  at  least  $762 
million  through  September  1989.  As  of  March  1991,  the  GAO  estimated  the  costs  had 
increased  to  at  least  $906  million  through  September  1991.  According  to  the  GAO,  the 
program  office  was  not  able  to  determine  how  much  of  the  increase  was  attributable  to 
providing  additional  system  capabilities  and  how  much  was  for  correcting  system 
deficiencies.  The  GAO  observed  that  costs  will  continue  to  increase,  since  the  Segment  will 
not  be  completed  until  July  1993. 

The  GAO  also  reported  that,  because  the  Command  and  Control  Segment  is  not  completely 
operational,  the  Air  Force  must  continue  to  maintain  the  older  Current  Data  System.  The 
GAO  reported  that  in  1988,  the  Air  Force  estimated  the  cost  to  operate  and  maintain  that 
System  was  $30  million  annually.  The  GAO  noted  that  the  Air  Force  did  not  provide  a 
revised  estimate  for  System  costs  during  its  current  review,  but  said  maintenance  costs 
should  fall,  since  parts  of  the  System  will  be  deactivated  by  the  end  of  1991.  (pp.  1-4, 
pp.  19-20/GAO  Draft  Report) 

DoD  Response:  Non-Concur.  While  some  of  the  basic  information  reported  is  correct,  the 
GAO  incorrectly  combines  acquisition  costs  associated  with  the  completed  development  of 
the  Command  and  Control  Segment  and  subsequent  life  cycle  costs  which  are  a  normal 
aspect  of  support  to  space  operations.  The  Command  and  Control  Segment  development 
phase  was  completed  in  1987.  Since  that  time,  the  Command  and  Control  Segment  has 
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supported  a  continually  evolving  mix  of  satellite  programs  that  have  new  or  changing 
missions  and  requirements.  Periodic  deliveries  of  new  and  modified  software  have  provided 
the  capabilities  needed  to  support  these  requirements,  as  well  as  the  maintenance  of  existing 
software.  The  costs  associated  with  this  continual  support  should  not  be  construed  as  cost  to 
"complete"  the  Command  and  Control  Segment  development. 

The  Air  Force  has  previously  provided  the  GAO  the  development  costs  associated  with  the 
Command  and  Control  Segment.  The  Congress,  in  the  FY  1989  Appropriations  Act  report 
language,  acknowledged  that  the  Command  and  Control  Segment  development  was 
complete,  and  found  the  on-going  software  support  to  be  "more  appropriately  an  operations 
and  maintenance  activity." 

•  FINDING  B:  The  Command  And  Control  Segment  Cannot  Perform  Key  Command 
And  Control  Operations.  The  GAO  explained  that  the  Command  and  Control  Segment  is 
required  to  be  able  to  contact  and  communicate  with  multiple  satellites  simultaneously- 
under  expected  peak  work  load  conditions.  The  GAO  found  that  the  contact  success  rate  has 
increased  since  1989.  The  GAO  reported,  however,  that  the  Air  Force  also  requires  that  each 
mission  control  complex  be  able  to  support  at  least  five  simultaneous  satellite  contacts.  The 
GAO  noted  that  this  is  a  key  requirement  that  is  used  to  judge  whether  the  Segment  is  ready 
to  handle  all  satellite  control  functions  at  a  mission  control  complex,  and  to  permit  the 
Current  Data  System  to  be  deactivated.  The  GAO  found,  however,  that  at  most  of  the 
control  complexes,  the  Command  and  Control  Segment  cannot  make  the  required  number  of 
simultaneous  contacts.  The  GAO  noted,  in  fact,  that  operators  sometimes  cut  down  on  the 
number  and  length  of  satellite  contacts  to  make  sure  they  did  not  exceed  system  capacity. 

The  GAO  found  that  the  Command  and  Control  Segment  also  has  other  serious  problems, 
such  as  long  recovery  time  from  system  failures.  According  to  the  GAO,  the  problems  are 
termed  critical  deficiencies  by  the  Air  Force,  because  they  must  be  fixed  before  the  Current 
Data  System  can  be  deactivated  and  the  Segment  turned  over  to  Air  Force  Space  Command. 
The  GAO  noted  that  some  problems  affect  all  satellites  and  control  complexes,  while  others 
affect  only  a  specific  satellite  or  complex. 

The  GAO  acknowledged  that  the  program  office  is  taking  positive  steps  to  correct  Command 
and  Control  Segment  critical  deficiencies.  The  GAO  reported,  for  example,  that  the  program 
office  is  (1)  trying  to  fix  software  problems,  (2)  adding  new  functions  in  software  releases, 
and  (3)  adding  equipment  to  three  mission  control  complexes  to  meet  capacity  and 
performance  requirements,  (pp.  3-4,  p.  19,  pp.  20-23/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  To  be  accurate,  a  distinction  must  be  made  between  those 
deficiencies  related  to  the  original  requirements  and  those  related  to  subsequently  evolved 
requirements  for  the  Command  and  Control  Segment. 

The  DoD  partially  concurs  with  the  GAO  assessment  of  simultaneous  contact  loading. 
Evolving  operational  requirements  have  resulted  in  increases  to  individual  contact  workloads 
-  which,  in  some  cases,  currently  prevent  a  five  simultaneous  contact  capability  under  present 
operational  conditions.  Contact  loading  requirements  in  the  Air  Force  Satellite  Control 
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Network  Mission  Control  Complexes  vary  from  three  to  six  simultaneous  contacts. 
Workloads  involved  in  each  satellite  contact  can  vary  significantly.  Most  Mission  Control 
Complexes  require  five  simultaneous  contacts,  but  each  has  a  unique  set  of  loading 
conditions  for  the  specified  number  of  contacts.  All  centers  meet  their  baseline  requirements 
for  Command  and  Control  System  transition  at  Onizuka  Air  Force  Base  and  for  the 
Consolidated  Space  Operations  Center  turnover  at  Falcon  Air  Force  Base.  Simultaneous 
contacts  at  the  specified  levels  routinely  are  accomplished  at  all  Mission  Control  Complexes. 
Additionally,  even  under  current  operational  conditions,  the  Command  and  Control  Segment 
supports  99  percent  of  Air  Force  Satellite  Control  Network  satellite  contacts  with  a  contact 
success  rate  in  excess  of  99  percent.  (The  required  contact  success  rate  is  95  percent.)  The 
system  has  supported  every  Air  Force  Satellite  Control  Network  satellite  launch  over  the  last 
year  and  the  Air  Force  Space  Command  currently  operates  all  operational  satellites  using  the 
system.  In  total  capacity,  during  Operation  Desert  Storm,  the  system  supported  130  percent 
of  the  total  1988  Air  Force  Satellite  Control  Network  satellite  contact  workload  it  was 
delivered  to  meet.  The  computer  systems,  which  were  replaced  by  the  Command  and 
Control  Segment,  are  almost  entirely  deactivated,  and  the  few  remaining  are  scheduled  to  be 
deactivated  by  June  1992.  Where  cunem  validated  requirements  for  capacity  greater  than 
now  available  exist,  modifications  to  improve  processing  efficiencies  and/or  increase 
capacity  have  been  designed  and  are  being  implemented  as  part  of  the  normal  sustaining 
engineering  support  process. 


•  FCHPINg  C;  Critical  Deficiencies  Are  Being  Resolved.  But  Hundreds  Remain.  The 
GAO  acknowledged  that,  since  its  1989  report,  the  Air  Force  corrected  several  hundred 
critical  deficiencies.  The  GAO  reported  that,  as  of  December  1990,  the  number  of  critical 
deficiencies  had  been  reduced  to  321,  as  compared  to  894  in  April  1989.  According  to  the 
GAO,  a  program  official  said  that  the  Air  Force  plans  to  fix  all  of  the  problems  by  the 
completion  of  the  Segment  upgrade  in  1993. 

The  GAO  also  pointed  out,  however,  that  correcting  the  deficiencies  has  taken  more  time 
than  the  Air  Force  expected.  The  GAO  cited  prior  Air  Force  estimates  for  resolving 
problems  that  were  not  met.  The  GAO  concluded,  therefore,  that  the  latest  estimate  for 
resolving  all  problems  may  not  be  realistic.  In  addition,  the  GAO  pointed  out  that  the  Air 
Force  continues  to  find  new  critical  deficiencies  in  each  new  software  release.  According  to 
the  GAO,  as  of  December  1990,  some  of  those  deficiencies  had  not  even  been  scheduled  to 
be  fixed.  Overall,  the  GAO  observed  that  slow  progress  on  some  of  these  critical  problems 
increases  the  risk  that  the  Command  and  Control  Segment  will  not  be  completed  on  time, 
(pp.  3-4,  p.  19,  pp.  23-25/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  Again,  the  issue  is  one  of  differentiating  between 
"deficiencies"  associated  with  the  development  phase  of  the  Command  and  Control  Segment 
and  those  that  are  the  result  of  subsequent  normal  operational  requirement  evolution.  The 
term  "critical  deficiencies,"  as  used  by  the  Air  Force  Satellite  Control  Network  operators  and 
program  office,  refers  to  "unsatisfied  operational  requirements  essential  to  support  of  a 
specified  event"  (e.g.,  a  launch,  elimination  of  a  former  mode  of  operating,  turnover  of  a  new 
capability,  ability  to  support  a  specific  mission).  Because  the  Air  Force  Satellite  Control 
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Network  requirements  are  continually  evolving  to  meet  new  operational  needs,  there  will 
likely  always  exist  a  set  of  currently  "critical  deficiencies."  The  term  cannot  and  should  not 
be  interpreted  to  be  synonymous  with  "flaw  in  delivered  product,"  as  the  GAO  apparently 
assumes.  Nor  does  it  necessarily  imply  a  condition  that  represents  a  risk  to  satellite 
operations.  As  an  example,  the  term  "turnover-critical  deficiency"  is  associated  with  a 
capability  or  performance  characteristic  mutually  agreed  to  be  a  condition  for  the  formal 
transfer  of  ownership  of  a  system  from  the  Air  Force  Systems  Command  to  the  Air  Force 
Space  Command. 

Many  of  the  214  requirements  stipulated  by  the  Air  Force  Space  Command  have  been 
determined  to  be  new  requirements,  outside  the  baseline  objectives  of  the  Consolidated 
Space  Operations  Center  program.  Those  requirements  will  be  prioritized  and  delivered 
when  appropriate,  as  part  of  the  ongoing  Air  Force  Satellite  Control  Network  improvement 
and  modernization  program. 

With  respect  to  formal  transfer  of  operational  Command  and  Control  Segment  systems 
within  the  Space  Operations  Complex  to  the  Air  Force  Space  Command,  the  last  six  software 
modifications  termed  "turnover-critical"  are  scheduled  for  delivery  in  June  1992. 

Formal  turnover  of  Mission  Control  Complexes  to  the  Air  Force  Space  Command  in  1993 
will  complete  a  carefully  planned  set  of  modification  and  maintenance  efforts  needed  to  meet 
operational  requirements  and  turnover  agreements,  as  well  as  thorough  operator  testing  and 
training. 

All  software  modifications  termed  "transition-critical”  (i.e.,  necessary  to  complete  the 
transfer  of  operations  at  Consolidated  Space  Test  Center  off  of  the  old  data  system)  have 
been  delivered.  As  previously  stated,  the  computer  systems  which  were  replaced  by  the 
Command  and  Control  Segment  are  almost  entirely  deactivated,  and  the  few  remaining  are 
scheduled  to  be  deactivated  by  June  1992. 

With  99  percent  of  all  Air  Force  Satellite  Control  Network  satellite  contact  activity  requiring 
data  analysis  presently  supported  by  the  Command  and  Control  Segment  at  over  99  percent 
mission  success  rate,  and  with  a  total  workload  presently  exceeding  130  percent  of  that 
which  the  original  system  was  delivered  to  meet,  significant  deficiencies  associated  with  the 
development  phase  are  small  in  number  and  are  not  deemed  critical  to  continued  operations 
or  to  turnover.  The  DoD  concurs  that  delivery  of  system  modifications  to  correct  some 
"critical  deficiencies"  has  taken  more  time  than  originally  expected  in  1987.  The  delivery  of 
needed  capability  is  scheduled  based  on  priorities  established  by  the  operational  users  of  the 
systems.  Significant  budget  reductions  in  FY  1988,  FY  1989,  and  again  in  FY  1991,  caused 
severe  disruption  of  program  plans.  The  workload  associated  with  higher  priority 
requirements  (e.g.,  to  support  launch  of  new  satellites)  also  contributed  to  the  delay. 
However,  transition  of  operations  from  the  old  data  system  has  proceeded  on  or  ahead  of  the 
schedule  established  in  FY  1989.  Transfer  of  the  satellite  operations  to  the  Air  Force  Space 
Command  is  on  schedule,  the  system  has  always  met  the  need  dates  for  launch,  and  the 
turnover  to  the  Air  Force  Space  Command  of  operational  Mission  Control  Complexes  is 
scheduled  consistent  with  current  program  direction.  As  previously  stated,  the  term 
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"completion"  is  inappropriate  for  the  continuing  evolution  of  this  system  in  response  to 
operational  needs. 

•  FINDING  D;  Command  And  Control  Segment  Requirements  Are  Not  Adequately 
Defined.  The  GAO  reported  that  the  Command  and  Control  Segment  contract  requires 
compliance  with  Military  Standards  483A  and  490A,  which  call  for  detailed  requirements  for 
computer  systems,  including  peak  processing  work  loads.  The  GAO  explained  that  the  work 
loads  are  critical  in  assuring  that  the  system  will  function  effectively  under  maximum 
expected  stress,  and  to  help  size  the  system,  develop  stress  tests,  and  serve  as  input  for 
capacity  and  performance  management. 

In  the  case  of  the  Command  and  Control  Segment,  the  GAO  observed  the  requirements  are 
defined  by  work  load  scenarios  that  describe  various  factors,  such  as  (1)  the  mix  of  satellites 
and  types  of  contacts  that  must  be  controlled  simultaneously,  (2)  the  number  of  terminals  the 
system  must  support,  and  (3)  the  amount  of  data  to  be  processed.  The  GAO  further  observed 
that  these  scenarios  should  include  those  that  place  the  greatest  expected  demand  on  system 
resources. 

The  GAO  found,  however,  that  the  Air  Force  last  updated  the  expected  peak  work  loads  for 
the  control  complexes  in  1987.  The  GAO  pointed  out  that  since  that  time,  the  number  of 
satellites  has  increased  by  36  percent,  and  the  complexes  are  handling  different  types  and 
numbers  of  satellites.  The  GAO  concluded,  therefore,  that  the  1987  expected  peak  work 
loads  are  obsolete.  The  GAO  further  concluded  that,  without  updated  requirements,  the  Air 
Force  cannot  (1)  assure  that  the  system  is  properly  sized.  (2)  develop  tests  to  adequately 
stress  the  system,  and  (3)  reliably  assess  the  impact  of  continual  Segment  changes  on  mission 
effectiveness.  The  GAO  concluded  that  this  problem,  together  with  the  other  three  problems 
identified  (see  Findings  E,  F,  and  G),  raise  serious  doubts  not  only  about  the  ability  of  the 
Air  Force  to  complete  the  Command  and  Control  Segment,  but  also  about  the  capability  of 
the  Segment  to  meet  its  mission  requirements,  (p.  5,  pp.  26-28/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  The  Command  and  Control  Segment  acquisition  fully 
complied  with  Military  Standards  483A  and  490A.  The  requirements  for  the  system  were 
fully  documented  in  a  hierarchical  set  of  baseline  requirements  documents  which  included  an 
extensively  detailed  Appendix  to  the  Baseline  Requirements  Specification  that  covered  peak 
loading  requirements. 

The  GAO  correctly  states  that  requirement  changes,  which  have  occurred  subsequent  to  the 
completion  of  the  development  phase  in  1987,  have  not  been  incorporated  in  the  Baseline 
Requirements  Specification.  The  Air  Force  Satellite  Control  Network  requirements  control 
process,  used  since  1987,  has  updated  appendices  to  the  baseline  requirements  specific  tion. 
the  allocated  baseline,  lower  level  design,  and  those  product  specifications  considered 
adequate  for  support  of  the  operational  system. 

The  Command  and  Control  Segment  was  shown  to  be  contractually  compliant  with  the 
Baseline  Requirements  Specification  at  the  completion  of  the  Data  System  Modernization 
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program.  The  subsequent  increase  in  satellite  programs  and  network  loading  may  very  well 
have  changed  peak  loading  requirements  identified  in  the  development  contract.  As 
indicated  in  the  DoD  response  to  Finding  B,  the  Command  and  Control  Segment  presently 
supports  a  satellite  mix  and  associated  network  workload  far  in  excess  of  the  levels  present 
upon  completion  of  the  development  phase  of  the  program.  Given  the  mission  success  rates 
associated  with  such  support,  the  GAO  conclusion  that  there  are  serious  doubts  the 
Command  and  Control  Segment  can  meet  its  mission  requirements  is  unwarranted.  As 
previously  indicated,  the  Air  Force  has  the  mechanisms  in  place  to  identify  and  meet  those 
requirements  necessary  for  turnover. 

•  FIND  IMS. El  The  Command  And  Control  Segment  Testing  Is  Inadequate,  The  GAO 
observed  that  expected  peak  work  loads  are  not  only  critical  to  developing  a  system,  but  are 
essential  for  testing  a  system  and  its  modifications  to  determine  if  the  system  can  do  its  job. 
The  GAO  further  observed,  however,  that  until  the  mission  control  complexes  have  updated 
work  load  requirements,  the  Air  Force  cannot  (1)  operationally  test  the  current  Command 
and  Control  Segment  configuration  to  ensure  the  more  taxing  satellite  scenarios  can  be  met, 
and  (2)  stress  test  each  software  release  in  a  laboratory  setting  before  it  is  used  in  the 
complexes. 

The  GAO  observed  that  critical  deficiencies  continue  to  be  found  after  new  and  revised 
software  is  released  for  use.  According  to  the  GAO,  that  is  due,  at  least  in  part,  to  the  fact 
the  testing  laboratory  is  not  equipped  to  simulate  the  high  stress  work  loads  the  control 
complexes  must  support.  The  GAO  reported  that,  between  March  1989  and  March  1990, 
three  major  software  releases  were  tested  at  the  laboratory,  but  no  new  critical  deficiencies 
were  found.  After  the  releases  were  used  in  the  control  complexes,  however,  the  GAO 
reported  that  about  175  new  critical  deficiencies  were  identified.  The  GAO  concluded  that 
testing  limitations,  together  with  the  other  three  problems  identified  (see  Findings  D,  F,  and 
G),  raise  serious  doubts  not  only  about  the  ability  of  the  Air  Force  to  complete  the  Command 
and  Control  Segment,  but  also  about  the  capability  of  the  Segment  to  meet  its  mission 
requirements,  (pp.  5-6,  p.  26,  pp.  28-31/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  In  1988,  Headquarters,  Air  Force  Systems  Command 
performed  an  independent  review  of  the  Command  and  Control  Segment  and  determined  that 
additional  testing  capabilities  would  be  beneficial  for  managing  the  sustaining  engineering 
activities.  An  additional  test  facility  was  defined  to  meet  validated  test  requirements.  Within 
resources  provided,  additional  test  capabilities  were  activated  in  1990. 

The  DoD  agrees  that  the  current  test  tools  available  do  not  allow  testing  at  peak  work  load 
conditions.  The  Air  Force  obviously  would  prefer  to  have  a  higher  fidelity  test  capability, 
but  existing  budgets  have  not  allowed  that  level  of  upgrade.  Requirements  are  currently 
identified  to  upgrade  facilities  and  test  capability  improvements  will  compete  for  funding  in 
the  budget  cycle. 
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•  FINDING  F:  The  Command  And  Control  Segment  Capacity  And  Performance 
Management  Program  Is  Inadequate.  The  GAO  reported  that  Federal  regulations  require 
agencies  to  manage  computer  capacity  and  performance.  To  do  so,  the  GAO  observed  that 
managers  must  monitor  system  performance  and  capacity  utilization  regularly.  The  GAO 
further  observed  that  agencies  should  routinely  collect  and  analyze  detailed  capacity  and 
performance  data,  which  would  also  help  predict  future  needs. 

The  GAO  found,  however,  that  the  Air  Force  has  not  set  up  an  effective  capacity  and 
performance  management  program.  Instead,  the  GAO  found  that  the  program  office  relies 
on  three  sources  of  data  to  manage  Segment  resources:  (1)  data  on  satellite  success  rates, 
using  the  Current  Data  System  and  the  Command  and  Control  Segment;  (2)  the  perception  of 
computer  operators  of  Segment  limitations;  and  (3)  infrequent  ad  hoc  analyses  of  computer 
capacity  and  performance.  The  GAO  pointed  out,  however,  that  such  sources  provide  only 
indications  of  system  performance,  but  do  not  measure  actual  use  or  reliably  assess 
performance  on  a  continuing  basis.  In  addition,  the  GAO  observed  that  these  sources  do  not 
offer  the  careful,  comprehensive  analysis  needed  to  manage  a  system  as  large  and  complex 
as  the  Command  and  Control  Segment.  The  GAO  concluded  that,  since  the  Air  Force  does 
not  have  an  effective  capacity  and  performance  management  program,  it  does  not  know  how 
well  the  Segment  is  performing  and  cannot  determine  how  much  more  capacity,  if  any,  is 
needed  to  meet  ever  changing  future  requirements. 

The  GAO  found  that  the  program  office  is  not  planning  to  improve  capacity  and 
performance  management  in  the  near  future.  According  to  the  GAO,  the  program  office 
cited  limited  funds  and  the  need  for  the  contractor  to  first  solve  Segment  software  and 
hardware  problems  as  reasons  for  not  approving  a  sustaining  engineering  contractor  request 
to  measure  Segment  performance  within  the  mission  control  complexes.  On  the  other  hand, 
the  GAO  found  that  in  March  1990  the  Second  Space  Wing  reorganized  its  computer 
performance  branch  to  place  more  emphasis  on  performance  and  capacity  planning  and 
analysis.  The  GAO  termed  it  a  constructive  step  towards  assessing  performance  and 
capacity.  The  GAO  concluded  that  inadequate  Command  and  Control  Segment  capacity  and 
performance  management,  together  with  the  other  three  problems  identified  (see  Findings  D, 
E,  and  G),  raise  serious  doubts  not  only  about  the  ability  of  the  Air  Force  to  complete  the 
Segment,  but  also  about  the  capability  of  the  Segment  to  meet  its  mission  requirements,  (p. 
7,  p.  26,  pp.  31-36,  pp.  41-42/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  There  currently  exists  no  formal  capacity  and 
performance  management  program  for  the  Command  and  Control  Segment.  To  date,  the  Air 
Force  has  found  its  current  method  of  assessing  capacity  and  performance  to  be  effective  for 
operational  and  planning  support.  System  performance  in  operation  is  constantly  monitored 
by  mission  sustaining  engineering  support.  Performance  upgrades,  like  other  requirements, 
are  validated  by  operations  organizations  as  mission  workloads  indicate  they  are  needed. 
The  Air  Force  Space  Command  continually  evaluates  data  collected  by  Air  Force  Satellite 
Control  Network  program  office  engineers  and  by  Air  Force  Space  Command  and  other 
organizations'  operators.  An  example  of  a  mission  change,  which  directed  planning  and 
execution  of  improvements  to  capacity  and  performance  is  the  1988  requirement  to  support 
two  additional  Defense  Support  Program  satellites.  That  satellite  has  a  very  high  telemetry 
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processing  workload.  Engineering  analysis  identified  a  solution  using  state-of-the-art 
workstation  technology  in  conjunction  with  basic  Command  and  Control  Segment 
Capabilities  would  be  the  most  effective.  That  new  capability  is  now  undergoing  final 
testing  for  delivery  to  operations. 

The  Federal  regulations  cited  by  the  GAO  are  general  guidelines  for  acquiring  and  managing 
computer  systems  subject  to  the  Brooks  Act  (40  USC  1059).  The  Command  and  Control 
Segment  is  exempted  from  the  Brooks  Act  by  the  Nunn-Wamer  Amendment  (10  USC 
2315),  which  provides  separate  standards  for  mission  critical  computer  resources. 


•  FINDING  G:  Hit  Command  And  Control  Segment  Software  Documentation  Is  Not 
Adequate.  The  GAO  reported  that  a  fourth  problem  that  has  affected  the  ability  of  the  Air 
Force  to  upgrade,  test,  modify,  and  maintain  the  Command  and  Control  Segment  is  poor 
software  documentation.  The  GAO  explained  that  software  documentation  should  be 
complete,  up-to-date,  and  well  organized— otherwise,  software  maintenance  takes  more  time 
and  effort,  and  there  is  less  assurance  that  software  modifications  will  function  as  required. 

The  GAO  found,  however,  that  the  Command  and  Control  Segment  software  documentation 
is  incomplete,  out-of-date,  and  difficult  to  use.  According  to  the  GAO,  as  of  April  1991,  the 
Air  Force  Operational  Test  and  Evaluation  Center  had  evaluated  12  of  28  Segment  software 
configuration  items  and  found  that  most  were  less  than  acceptable.  The  GAO  reported  that 
the  testing  officials  said  that  inadequate  Segment  documentation  represents  a  significant  risk 
to  satellite  control  operations,  since  it  impairs  the  ability  to  identity  and  correct  problems  in  a 
timely  and  cost  effective  way.  The  GAO  reported  that  Air  Force  officials  attributed  the 
inadequate  software  documentation  to  (1)  poor  management  control  over  the  initial  Segment 
development,  and  (2)  an  overriding  interest  in  getting  the  system  within  budget  and  on 
schedule. 

The  GAO  pointed  out  that  prior  audits  and  studies  have  shown  repeatedly  that  poor  software 
documentation  increases  maintenance  costs,  which  comprise  the  large-t  percentage  of  system 
life  cycle  costs.  The  GAO  concluded  that  unless  the  Air  Force  corrects  deficiencies  in  the 
Segment  documentation,  it  will  be  increasingly  difficult  to  maintain  and  enhance  the 
Segment  as  requirements  continue  to  change  and  evolve,  and  the  costs  to  maintain  the 
Segment  are  likely  to  continue  to  increase.  The  GAO  also  concluded  that  inadequate 
Command  and  Control  Segment  software  documentation,  together  with  the  other  three 
problems  identified  (Findings  D,  E,  and  F),  raise  serious  doubts  not  only  about  the  ability  of 
the  Air  Force  to  complete  the  Segment,  but  also  about  the  capability  of  the  Segment  to  meet 
its  mission  requirements,  (p.  8,  p.  26,  pp.  36-40,  p.  42/GAO  Draft  Report) 

DoD  Response:  Partially  concur.  The  Air  Force  Operational  Test  and  Evaluation  Center 
has  identified  certain  portions  of  existing  documentation  that  they  rate  "less  than  acceptable" 
in  that  these  portions  do  not  meet  certain  threshold  standards.  The  fact  that  some  of  the 
Command  and  Control  Segment  documentation  does  not  meet  the  thresholds  is  not 
interpreted  by  the  Air  Force  Operational  Test  and  Evaluation  Center  or  by  the  Air  Force 
Space  Command  as  indicating  unacceriability  for  turnover.  The  system  is  entirely 
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supportable,  especially  in  modem  software  engineering  support  environments,  with  the 
present  documentation. 

The  Air  Force  Operational  Test  and  Evaluation  Center  reports  did  not  identify  any  failures  to 
comply  with  applicable  regulations  or  standards.  They  reflect  an  evolving  understanding  of 
good  engineering  practices  -  the  twelve  reports  cited  by  GAO  used  a  different  set  of 
standards  in  most  evaluations.  Therefore,  program  office  responses  in  terms  of  actions  on 
each  point  of  the  published  evaluation  were  not  appropriate.  However,  many  beneficial 
ideas  were  taken  from  the  Air  Force  Operational  Test  and  Evaluation  Center  reports  and 
have  been  incorporated  into  the  Command  and  Control  Segment  documentation  of  new 
requirements  and  maintenance  changes.  The  program  office  has  worked  closely  with  the  Air 
Force  Space  Command,  the  Air  Force  Operational  Test  and  Evaluation  Center,  and  the 
Command  and  Control  Segment  support  contractors  to  identify  and  implement 
improvements  to  many  documents  as  part  of  the  normal  maintenance  of  the  system.  In 
addition,  the  Air  Force  Satellite  Control  Network  is  increasing  the  use  of  development  and 
testing  tools,  such  as  the  Jovial  reverse  engineering  tool,  which  was  identified  by  some  Air 
Force  Operational  Test  and  Evaluation  Center  reports  as  a  significant  factor  in  their 
evaluation  but  which  could  not  be  considered  at  the  time  of  the  evaluations.  Thus,  while  the 
Command  and  Control  Segment  has  always  met  required  standards,  the  Air  Force  has  sought 
ways  to  economically  improve  processes  and  operational  systems.  The  evolving  criteria 
used  by  the  Air  Force  Operational  Test  and  Evaluation  have  all  been  considered  and  have 
been  incorporated  in  Command  and  Control  Segment  modifications  wherever  possible. 

The  FY  1990  Appropriations  Act  included  Senate  Armed  Services  Committee  language, 
which  expressed  concerns  about  the  adequacy  of  documentation  —  partially  in  the  context  of 
its  sufficiency  to  support  competition  for  the  award  of  a  new  sustaining  engineering  contract. 
The  Air  Force  Competition  Advocate  General  evaluated  the  preparations  for  competition, 
including  documentation,  in  January  1991,  and  determined  there  were  no  deficiencies  that 
precluded  fair  competition. 


RECOMMENDATIONS 


•  RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Air  Force  to  ensure  that  the  Command  and  Control  Segment  work  load 
requirements  (specifically,  the  peak  work  load  requirements  for  each  of  the  mission  control 
complex  computer  systems)  are  immediately  updated  and  kept  current,  (p.  8,  p.  42/GAO 
Draft  Report) 

DoD  Response:  Partially  concur.  The  Air  Force  has  defined  workload  requirements  for 
turnover  of  operational  Mission  Control  Complexes  to  the  Air  Force  Space  Command  in 
compliance  with  DoD  regulations.  The  Air  Force  will  continue  to  define  and  evaluate 
evolving  peak  workload  requirements  for  all  Air  Force  Satellite  Control  Network  Mission 
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Control  Complexes  as  new  missions  are  added.  This  information  will  be  maintained  in 
appropriate  system  documentation. 

#  RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Air  Force  to  ensure  that  the  updated  work  load  requirements  are  used  to 
(1)  operationally  test  the  Command  and  Control  Segment  at  the  icissior  control  complexes 
before  the  complexes  are  turned  over  to  the  Space  Command,  and  (2)  stress  test  each 
Command  and  Control  Segment  software  release  before  it  is  used  in  the  mission  control 
complexes,  (p.  8,  pp.  4243/GAO  Draft  Report) 

See  comment  9.  DoD  Response:  Partially  concur.  The  System  Program  Office  and  the  Air  Force  Space 

Command  are  identifying  those  tests  necessary  to  validate  the  Command  and  Control 
Segment  for  operational  turnover  to  the  Air  Force  Space  Command,  consistent  with 
Consolidated  Space  Operations  Center  program  requirements.  That  effort  is  expected  to  be 
completed  by  December  1991  with  issuance  of  the  updated  Consolidate  Space  Operations 
Center  Test  and  Evaluation  Master  Plan.  While  the  requirements  do  not  necessarily 
represent  current  peak  workloads  in  ail  Mission  Control  Complexes,  the  Air  Force  Satellite 
Control  Network  Improvement  and  Modernization  Program  will  ensure  that  modifications  to 
the  Command  and  Control  Segment  are  made,  as  necessary,  to  meet  evolving  operational 
workload  requirements.  The  Air  Force  recognizes  that  current  software  test  tools  do  not 
have  the  capability  to  stress  test,  at  peak  workloads,  each  Command  and  Control  Segment 
software  release  before  it  is  used  in  the  Mission  Control  Complexes.  As  indicated  in  the 
DoD  response  to  Finding  E,  the  Air  Force  has  identified  requirements  to  upgrade  the  test 
environment,  and  these  requirements  will  compete  for  funding  with  other  requirements  in  the 
budget  cycle. 

#  RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Air  Force  to  ensure  that  a  comprehensive  Command  and  Control 
Segment  capacity  and  performance  management  program  is  immediately  established,  (p.  8, 
pp.  4243/GAO  Draft  Report) 

See  comment  10  DoD  Response:  Partially  concur.  Current  methods  are  sufficient  for  tracking  and  managing 

the  operational  system.  However,  the  Air  Force  will  assess  the  capabilities  in  place  and  will 
determine  what  steps  should  be  taken  to  improve  capacity  and  performance  management. 
That  analysis  is  to  be  completed  by  August  1992. 

#  RECOMMENDATION  4:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Air  Force  to  ensure  that  the  Command  and  Control  Segment  software 
documentation  deficiencies  are  corrected  before  the  system  is  turned  over  to  the  Space 
Command,  and  that  documentation  is  adequately  prepared  and  maintained  in  the  future, 
(p.  8,  pp.  4243/GAO  Draft  Report) 
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See  comment  1 1 . 


DoD  Response:  Partially  concur.  Although  some  of  the  earlier  documentation  does  not 
satisfy  some  of  the  Air  Force  Operational  Test  and  Evaluation  Center  criteria  fully,  the 
documentation  complies  with  all  applicable  regulations  and  standards  and  is  fully  successful 
in  meeting  day-to-day  maintenance  and  upgrade  needs.  The  program  office  incorporated 
many  of  the  Air  Force  Operational  Test  and  Evaluation  Center  recommendations  into  the 
documentation  together  with  updates  to  incorporate  new  mission  requirements.  Process 
improvements  addressed  by  the  Air  Force  Operational  Test  and  Evaluation  Center,  such  as 
automation  of  documentation  and  testing  environments,  are  also  being  incorporated  into  Air 
Force  Satellite  Control  Network  activities,  completely  changing  the  basis  for  future 
evaluations.  The  Air  Force  will  ensure  updates  to  the  documentation  adequately  support  a 
competitive  contractor  maintenance  concept  as  required  by  program  direction. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  letter 
dated  November  4,  1991. 


GAO  Comments 


1.  dod  incorrectly  characterizes  the  cost  information  in  the  report,  which 
clearly  discloses  that  the  costs  include  those  for  developing  ccs  and 
those  for  the  sustaining  engineering  needed  to  complete  development 
and  to  satisfy  new  requirements.  Since  the  purpose  of  ccs  was  to  replace 
the  old,  out-of-date  cds,  we  believe  it  is  appropriate  to  report  all  costs 
until  all  original  requirements  for  ccs  have  been  completed  and  cds  is 
deactivated.  As  noted  in  the  report,  the  Air  Force  was  not  able  to  deter¬ 
mine  how  much  of  the  sustaining  engineering  costs  were  for  meeting 
original  requirements  and  how  much  were  for  new  requirements. 

2.  dod  partially  concurred  with  our  finding  that  ccs  cannot  perform  key 
operations.  Its  partial  nonconcurrence  appears  to  center  on  the  Air 
Force’s  position  that,  today,  ccs  is  supporting  virtually  all  scheduled  sat¬ 
ellite  contacts — a  position  that  was  clearly  recognized  in  our  draft 
report.  However,  DOD  agreed  that  most  mccs  cannot  make  five  simulta¬ 
neous  contacts  under  all  expected  work-load  scenarios.  We,  therefore, 
continue  to  believe  that  ccs  may  not  meet  the  five  simultaneous  contact 
requirement  at  turnover  in  July  1993,  particularly  as  work  loads  con¬ 
tinue  to  change  and  grow.  The  Air  Force  stated  that  the  mccs  require¬ 
ments  vary  from  three  to  six  simultaneous  contacts.  We  have  revised 
the  report  accordingly. 

3.  dod  partially  concurred  with  our  finding  that  critical  deficiencies  still 
remain,  and  concurred  with  our  finding  that  correcting  some  critical 
deficiencies  has  taken  longer  than  originally  expected.  However,  dod 
objected  to  what  it  characterized  as  our  combining  deficiencies  associ¬ 
ated  with  the  development  phase  and  those  that  are  the  result  of  subse¬ 
quent  normal  operational  requirement  evolution  and  it  asserts  that  the 
last  six  turnover  critical  deficiencies  are  scheduled  to  be  corrected  by 
June  1992. 

The  321  critical  deficiencies  we  identified  were  based,  as  the  report 
clearly  states,  on  (1)  the  Air  Force’s  criteria — deficiencies  that  must  be 
fixed  before  cds  can  be  deactivated  and  ccs  turned  over  to  Space  Com¬ 
mand,  and  (2)  the  latest  data  available  when  we  completed  our  field 
work.  We  analyzed  the  additional  information  the  Air  Force  recently 
supplied  to  support  its  assertion  that  the  last  six  critical  deficiencies  are 
scheduled  to  be  corrected  by  June  1992.  The  Air  Force’s  information 
was  inconsistent  with  the  information  previously  provided  and  the  Air 
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Force  did  not  provide  explanations  for  these  inconsistencies.  Further, 
the  Air  Force  did  not  provide  sufficient  information  for  us  to  determine 
the  disposition  of  any  of  the  321  critical  deficiencies — it  did  not  explain 
if  they  were  resolved,  reprioritized,  or  no  longer  considered  critical  to 
turnover  and  deactivation  of  CDS.  Finally,  the  Air  Force’s  latest  data 
showed  that  13  critical  deficiencies  had  not  even  been  scheduled  to  be 
corrected  by  June  1992.  Therefore,  we  did  not  revise  the  information  in 
the  report. 

4.  We  believe  that  the  actions  the  Air  Force  said  it  took  respond  to  our 
concerns,  as  we  explain  in  the  analysis  of  their  response  to  our  recom¬ 
mendation  (see  number  8).  dod’s  basis  for  only  partially  concurring  with 
our  finding  is  not  clear. 

5.  While  dod  partially  concurred  with  our  finding  that  ccs  testing  is 
inadequate,  it  was  difficult  to  determine  with  what  it  disagreed.  The  Air 
Force  agrees  that  its  current  test  tools  are  not  adequate  to  test  ccs  at 
peak  work-load  conditions.  Further,  the  Air  Force  recognizes  the  need  to 
upgrade  its  testing  capabilities. 

6.  dod  states  that  while  no  formal  capacity  and  performance  program 
exists,  the  Air  Force  has  found  its  current  method  of  assessing  capacity 
and  performance  to  be  effective.  We  disagree.  Our  analysis  showed  that 
the  Air  Force  does  not  have  a  method  to  measure  actual  computer  use  or 
reliably  assess  performance  on  a  regular  basis. 

Further,  dod  notes  that  the  federal  regulations  on  capacity  and  perform¬ 
ance  management  discussed  in  the  report  do  not  apply  to  ccs.  We  did  not 
mean  to  imply  that  the  Air  Force  is  required  to  follow  the  specific  regu¬ 
lations  we  cited  and  have  revised  the  report  to  eliminate  any  such  impli¬ 
cation.  The  regulations  and  guidance  were  cited  because  they  are  good 
examples  of  what  should  be  done.  We  believe  that  both  dod  and  the  Air 
Force  agree  on  the  main  point — every  system,  especially  one  the  size 
and  complexity  of  ccs,  should  have  a  capacity  and  performance  manage¬ 
ment  program  to  assure  that  information  is  processed  efficiently,  effec¬ 
tively,  and  economically. 

7.  dod  partially  concurred  with  our  finding  on  documentation,  stating 
that  afutec  found  that  some  documentation  is  less  than  acceptable,  but 
that  this  is  not  interpreted  by  apcttec  or  by  Space  Command  as  indi¬ 
cating  that  the  documentation  is  unacceptable  for  turnover.  We  disagree 
with  this  interpretation,  apotec’s  evaluation  found  that  8  of  the  12 
software  configuration  items  it  evaluated  had  documentation  that  would 
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make  it  difficult  to  maintain  the  software  and  that  some  documentation 
needs  to  be  changed  before  it  can  be  used  as  an  effective  maintenance 
tool,  afutec’s  conclusion — that  the  poor  documentation  would  make  the 
system  difficult  to  maintain  and  that  improvements  were  needed — 
makes  it  clear  that  portions  of  the  documentation  are  unacceptable  for 
turnover. 

Further,  dod  cited  the  Air  Force’s  Competition  Advocate  General’s  eval¬ 
uation  of  the  competition  for  the  new  sustaining  engineering  contract  as 
support  for  ccs’  software  documentation  being  adequate.  We  discussed 
the  evaluation  with  the  Competition  Advocate  General  and  do  not 
believe  his  evaluation  is  relevant  to  our  finding.  The  Competition  Advo¬ 
cate  General  said  that  he  did  not  independently  evaluate  the  documenta¬ 
tion  nor  did  he  review  apotec’s  findings.  To  evaluate  the  documentation, 
he  discussed  its  adequacy  with  one  potential  bidder.  His  conclusion  that 
there  were  no  documentation  deficiencies  that  precluded  fair  competi¬ 
tion  is  entirely  different  from  any  assessment  of  how  the  documentation 
will  help  in  maintaining  the  software. 

8.  dod  concurred  with  our  recommendation  to  update  and  keep  current 
work  load  requirements,  dod  stated  that  the  Air  Force  has  defined  work¬ 
load  requirements  for  turnover  of  the  mccs  and  that  it  will  define  and 
continue  to  evaluate  evolving  peak  work-load  requirements  for  all  mccs 
as  new  missions  are  added  dod  stated  that  the  Air  Force  had  updated 
the  requirements,  but  it  did  not  provide  this  information  to  us.  There¬ 
fore,  we  were  not  able  to  evaluate  the  adequacy  of  the  updated 
requirements. 

9.  dod  partially  concurred  with  our  second  recommendation,  dod  stated 
that  the  Air  Force  is  identifying  the  tests  necessary  to  validate  ccs  for 
operational  turnover  to  Space  Command  consistent  with  satellite  pro¬ 
gram  requirements,  dod  noted  that  while  these  requirements  do  not  nec¬ 
essarily  represent  current  peak  work  loads,  the  Air  Force  will  ensure 
that  modifications  to  ccs  are  made  as  necessary  to  meet  evolving  work¬ 
load  requirements,  dod’s  response  recognizes  the  need  to  ensure  that  the 
updated  work-load  requirements  are  used  to  operationally  test  ccs  at 
mccs  before  they  are  turned  over  to  Space  Command. 

However,  while  dod  agreed  that  the  Air  Force’s  current  software  test 
tools  are  not  adequate  to  stress  test  ccs  at  peak  work  loads  before  it  is 
used  in  the  mccs,  dod  did  not  require  the  Air  Force  to  conduct  the  stress 
tests,  dod  stated  the  Air  Force  has  plans  to  upgrade  the  test  environ¬ 
ment,  but  noted  that  these  requirements  will  compete  for  funding  in  the 


Page  45 


GAO/IM TEC-92'3  Satellite  Control  System  Upgrade 


Appendix  n 

Comments  From  the  Department  of  Defense 


budget  cycle.  While  we  recognize  that  dod  and  the  Air  Force  are  under 
funding  constraints,  testing  is  extremely  important  to  assuring  that  ccs 
can  meet  its  mission.  In  addition,  correcting  problems  before,  instead  of 
after,  implementation  should  save  funds  over  the  long  run. 

10.  dod  partially  concurred  with  our  recommendation.  While  dod  stated 
that  the  current  methods  are  sufficient,  dod  also  directed  the  Air  Force 
to  assess  its  current  methods  and  determine  what  steps  should  be  taken 
to  improve  capacity  and  performance  management.  We  believe  that  the 
Air  Force’s  current  efforts  to  manage  ccs  do  not  provide  the  information 
necessary  to  adequately  manage  system  capacity  and  performance. 
However,  we  believe  that  getting  the  Air  Force  to  assess  its  current 
efforts  is  a  step  in  the  right  direction  and  should  demonstrate  that  it 
needs  a  capacity  and  performance  management  program  that  regularly 
monitors  system  performance  and  capacity  utilization  using  hardware 
and  software  monitors  and  analytical  modelling  tools. 

11.  dod  stated  that  it  partially  concurred  with  our  last  recommendation. 
DOD  also  stated  that  although  the  documentation  is  fully  successful  in 
meeting  day-to-day  maintenance  and  upgrade  needs,  it  will  require  the 
Air  Force  to  ensure  the  documentation  adequately  supports  a  competi¬ 
tive  contractor  maintenance  concept.  We  disagree  that  the  documenta¬ 
tion  is  adequate  for  maintaining  the  software,  apotec’s  evaluation 
concluded  that  the  documentation  will  make  it  difficult  to  maintain  the 
software  and  that  some  portions  will  need  changes  before  the  documen¬ 
tation  can  be  used  as  an  effective  maintenance  tool.  AFurec’s  conclusion 
makes  it  clear  that  portions  of  the  documentation  are  unacceptable  for 
turnover.  Therefore,  we  believe  dod  needs  to  take  another  look  at  this 
issue. 
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